08_800w_web1 (larbin2.6.3@unspecified.mail)
123metaspider-Bot (Version: 1.04, powered by www.123metaspider.com)
13TABS/13TABS-1.14 (https://www.13tabs.com/aboutUs.jsp)
1470.net crawler
20090717_aa_page (larbin2.6.3@unspecified.mail)
2Bone_LinkChecker/1.0 libwww-perl/6.03
2Bone_LinkChkr/1.0 libwww-perl/6.03
2ip.ru CMS Detector (http://2ip.ru/cms/)
2ip.ru CMS Detector (https://2ip.ru/cms/)
360spider-image
404 Checker [http://www.404checker.com/user-agent]
449 Overture-WebCrawler/3.8/Fresh (atw-crawler at fast dot no; http://fast.no/support/crawler.asp
50.nu/0.01 ( +http://50.nu/bot.html )
50.nu/0.01 ( http://50.nu/bot.html )
5acd734a-1ed3-4a17-ad9d-0b09ae7bc2d2/Nutch-1.7
777Siters/1.05 (+http://777s.ru/siters/)
7Siters/1.07 (+http://7ooo.ru/siters/)
7Siters/1.07 (+https://7ooo.ru/siters/)
8bo Crawler Bot
<a href='http://www.unchaos.com/'> UnChaos </a> From Chaos To Order Hybrid Web Search Engine.(vadim_gonchar@unchaos.com)
@LinkArchiver twitter bot
A6-Indexer
ADmantX Platform Semantic Analyzer - ADform - ADmantX Inc. - www.admantx.com - support@admantx.com
ADmantX Platform Semantic Analyzer - ADmantX Inc. - www.admantx.com - support@admantx.com
ADmantX Platform Semantic Analyzer - APAC - ADmantX Inc. - www.admantx.com - support@admantx.com
ADmantX Platform Semantic Analyzer - Turn - ADmantX Inc. - www.admantx.com - support@admantx.com
ADmantX Platform Semantic Analyzer US - Turn - ADmantX Inc. - www.admantx.com - support@admantx.com
ADmantX Platform Semantic Analyzer US Async - ADmantX Inc. - www.admantx.com - support@admantx.com
AESOP_com_SpiderMan
AFB/3.0 (+http://allloadin.com)
AHC/1.0
AHC/2.0
AISearchBot (Email: aisearchbot@gmail.com; If your web site doesn't want to be crawled, please send us a email.)
APIs-Google (+https://developers.google.com/webmasters/APIs-Google.html)
APIs-Google; (+https://developers.google.com/webmasters/APIs-Google.html)
ARC Reader (http://arc.semsol.org/)
Aboundex/0.2 (http://www.aboundex.com/crawler/)
Aboundex/0.3 (http://www.aboundex.com/crawler/)
AboutUsBot
AboutUsBot/Harpy (Website Analysis; http://www.aboutus.org/Aboutus:Bot; help@aboutus.org)
Abrave Spider v4 Robot 1 (http://robot.abrave.co.uk)
Abrave Spider v4 Robot 2 (http://robot.abrave.co.uk)
Abrave Spider v5.3 Robot 2 (http://robot.abrave.com)
Abrave v5.5 (http://robot.abrave.co.uk)
Abrave v6.0 (http://robot.abrave.com)
Accoona-AI-Agent/1.1.1 (crawler at accoona dot com)
Accoona-AI-Agent/1.1.2 (aicrawler at accoonabot dot com)
Accoona-Biz-Agent/1.1.1 crawler@accoona.com
AcoonBot/4.10.5 (+http://www.acoon.de)
AcquiaCrawler,gzip(gfe),gzip(gfe)
AcquiloSpider/1.0,gzip(gfe),gzip(gfe)
Ad Muncher v4.93.33707/4957
AdAuth/1.0; +https://www.adauth.com
Adaxas Spider (http://www.adaxas.net/)
AddThis.com (http://support.addthis.com/)
AddThis.com robot tech.support@clearspring.com
Adknowledge Engage Crawler
AdnormCrawler www.adnorm.com/crawler
AdsBot-Google
AdsBot-Google ( http://www.google.com/adsbot.html)
AdsBot-Google (+http://www.google.com/adsbot.html)
AdsBot-Google-Mobile (+http://www.google.com/mobile/adsbot.html) Mozilla (iPhone; U; CPU iPhone OS 3 0 like Mac OS X) AppleWebKit (KHTML, like Gecko) Mobile Safari
AdsBot-Google-Mobile (+http://www.google.com/mobile/adsbot.html) Mozilla (iPhone; U; CPU iPhone OS 3 0 like Mac OS X) AppleWebKit/537.36 (KHTML, like Gecko) Mobile Safari/537.36
AdsBot-Google-Mobile-Apps
AdsTxtCrawler/1.0
AhrefsBot.Feeds v0.1; http://ahrefs.com/
Aleksika Spider/1.0 ( http://www.aleksika.com/)
Altresium/4.6 (+http://www.altresium.com/bot.html)
Amazon CloudFront
Amazon Route 53 Health Check Service; ref:xxxxxxxx-xxxx-xxxxxxxxx-xxxxxxxxxxxx; report http://amzn.to/xxxxxxx
Amazon-Route53-Health-Check-Service (ref b0eb04d5-cb5e-40e7-839b-558e52fc3f0d; report http://amzn.to/1vsZADi)
Amazon-Route53-Health-Check-Service (ref c96ad03c-acdf-48c6-8ec7-45806bf08123; report http://amzn.to/1vsZADi)
Amfibibot/0.07 (Amfibi Robot; http://www.amfibi.com; agent@amfibi.com)
AmorankSpider/0.1; +http://amorank.com/webcrawler.html
AnomaliesBot/0.06-dev (The Anomalies Network Search Spider; http://www.anomalies.net; info@anomalies.net)
Ant/Ant-Nutch-1.1 (Ant Nutch Crawler; http://www.ant.com; crawler@ant.com)
AntBot/1.0 (http://www.ant.com)
AntBot/Ant-Nutch-1.1 (Ant Nutch Crawler; http://www.ant.com; crawler@ant.com)
Apache-HttpClient/4.2.3 (java 1.5)
Apache-HttpClient/4.2.5 (java 1.5)
Apache-HttpClient/4.3.1 (java 1.5)
Apache-HttpClient/4.3.3 (java 1.5)
Apache-HttpClient/4.3.5 (java 1.5)
Apache-HttpClient/4.4.1 (Java/1.8.0_65)
Apache-HttpClient/4.5 (Java/1.8.0_51)
Apache-HttpClient/4.5 (Java/1.8.0_72)
Apache-HttpClient/4.5 (Java/1.8.0_74)
Apache-HttpClient/4.5.1 (Java/1.7.0_79)
Apache-HttpClient/4.5.1 (Java/1.8.0_40)
Apache-HttpClient/4.5.10 (Java/1.8.0_201)
Apache-HttpClient/4.5.2 (Java/1.8.0_151)
Apache-HttpClient/4.5.2 (Java/1.8.0_161)
Apache-HttpClient/4.5.2 (Java/1.8.0_181)
Apache-HttpClient/4.5.2 (Java/1.8.0_65)
Apache-HttpClient/4.5.2 (Java/1.8.0_72)
Apache-HttpClient/4.5.3 (Java/1.8.0_121)
Apache-HttpClient/4.5.3-SNAPSHOT (Java/1.8.0_152)
Apache-HttpClient/4.5.7 (Java/11.0.3)
Apache-HttpClient/UNAVAILABLE (java 1.4)
ApacheBench/2.3
Apexoo Spider 1.1
AppCodes crawler - looking for iOS app mentions. More info: support@appcodes.com. Robots.txt id: AppCodesCrawler
AppEngine-Google; ( http://code.google.com/appengine)
AppEngine-Google; (+http://code.google.com/appengine; appid: downforeveryoneorjustme)
AppEngine-Google; (+http://code.google.com/appengine; appid: example)
AppEngine-Google; (+http://code.google.com/appengine; appid: s~buzzstream-tools-hrd)
AppEngine-Google; (+http://code.google.com/appengine; appid: s~closure-compiler-hrd)
AppEngine-Google; (+http://code.google.com/appengine; appid: s~feedly-social)
AppEngine-Google; (+http://code.google.com/appengine; appid: s~snapchat-proxy)
AppEngine-Google; (+http://code.google.com/appengine; appid: s~virustotalcloud)
AppEngine-Google; (+http://code.google.com/appengine; appid: xbmacx)
AppManager RPT-HTTPClient/0.3-3E
AppSignalBot/1.0 (+https://appsignal.com)
AppStoreScraperZ/1 CFNetwork/889.9 Darwin/17.2.0 (x86_64)
AppleNewsBot
Arachni/v1.5.1
Arachnida Web Crawler
Aranea Web Corpora Project (Sk Crawl) (+http://unesco.uniba.sk/guest)
ArchitextSpider
ArchiveBox/0.5.3 (+https://github.com/ArchiveBox/ArchiveBox/)
ArchiveTeam ArchiveBot/20140807.01 (wpull 0.37a1) and not Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/36.0.1985.125 Safari/537.36
ArchiveTeam ArchiveBot/20141009.02 (wpull 0.1002a1) and not Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/38.0.2125.101 Safari/537.36
ArchiveTeam ArchiveBot/20170106.02 (wpull 2.0.2)
ArchiveTeam ArchiveBot/20190617.01 (wpull 2.0.3) and not Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.2311.90 Safari/537.36
Argus/1.1 (Nutch; http://www.simpy.com/bot.html; feedback at simpy dot com)
ArielisBot/1.1 (+http://arielis.com/
Arikus_Spider
AskQuickly v2 (http://askquickly.org/)
Athena Site Analyzer (bestsiteanalyzer.com)
Automattic Analytics Crawler/0.1; http://wordpress.com/crawler/
Automattic Analytics Crawler/0.2; http://wordpress.com/crawler/
Automattic Analytics Crawler/0.2;+http://wordpress.com/crawler/
Autonomy Spider
AwarioRssBot/1.0 (+https://awario.com/bots.html; bots@awario.com)
AwarioSmartBot/1.0 (+https://awario.com/bots.html; bots@awario.com)
Axtaris Web Crawler/Axtaris-1.0.1 (Hello from Axtaris.org | Web-Crawler; http://axtaris.org; crawler at axtaris dot org)
B2B Bot
BIGLOTRON (Beta 2;GNU/Linux)
BLEXBot
BLP_bbot/0.1
BOT/0.1 (BOT for JCE)
BT Crawler 1.0,gzip(gfe),gzip(gfe),gzip(gfe)
BTWebClient/180B(9704)
BUbiNG (+http://law.di.unimi.it/BUbiNG.html)
BUbiNG - Research at Brno University of Technology - KNOT group - http://knot.fit.vutbr.cz/crawling/ - Stop? http://law.di.unimi.it/BUbiNG.html
BabalooSpider/1.3 (BabalooSpider; http://www.babaloo.si; spider@babaloo.si)
Backlink-Ceck.de (+http://www.backlink-check.de/bot.html)
Backlink-Check.de (+http://www.backlink-check.de/bot.html)
BacklinkCrawler (http://www.backlinktest.com/crawler.html)
BacklinkCrawler V (http://www.backlinktest.com/crawler.html)
Bad Neighborhood Header Detector (http://www.bad-neighborhood.com/header_detector.php)
Bad-Neighborhood Link Analyzer (http://www.bad-neighborhood.com/)
Baidu-YunGuanCe-Bot(ce.baidu.com)
Baidu-YunGuanCe-PerfBot(ce.baidu.com)
Baidu-YunGuanCe-SLABot(ce.baidu.com)
Baidu-YunGuanCe-ScanBot(ce.baidu.com)
Baidu-YunGuanCe-VSBot(ce.baidu.com)
BaiduImagespider ( http://help.baidu.jp/system/05.html)
Baiduspider+(+http://help.baidu.jp/system/05.html)
Baiduspider+(+http://www.baidu.com/search/spider.htm)
Baiduspider+(+http://www.baidu.com/search/spider_jp.html)
Baiduspider+(+http://www.baidu.jp/spider/)
Baiduspider-image+(+http://www.baidu.com/search/spider.htm)
Baiduspider-image+(+http://www.baidu.com/search/spider.htm)\nReferer: http://image.baidu.com/i?ct=503316480&z=0&tn=baiduimagedetail
Baiduspider-testbranch
BarkRowler/0.7 (+http://www.exensa.com/crawling)
Barkrowler/0.5.1 (experimenting / debugging - sorry for your logs ) http://www.exensa.com/crawl - admin@exensa.com -- based on BuBiNG
Barkrowler/0.7 (+http://www.exensa.com/crawl)
Barkrowler/0.9 (+http://www.exensa.com/crawl)
BarraHomeCrawler (albertof@barrahome.org)
BebopBot/2.5.1 (compatible; media crawler V1; http://www.apassion4jazz.net/bebopbot.html;)
BeijingCrawler
BejiBot Crawler (BNL Services; http://www.bejjan.net/crawler/)
Better Uptime Bot Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.169 Safari/537.36
Bidtellect/0.0.346.0
Bidtellect/0.0.643.0
BillyBobBot/1.0 (+http://www.billybobbot.com/crawler/)
Bimbot/1.0
BinGet/1.00.A (http://www.bin-co.com/php/scripts/load/)
BizwikiBot/0.3 (http://www.bizwiki.com/bizwikibot.htm)
BlackBerry9000/4.6.0.167 Profile/MIDP-2.0 Configuration/CLDC-1.1 VendorID/102 ips-agent
Blackboard Safeassign
Blackboard Safeassign/0.1 (a Storm-based Blackboard Safeassign web-crawler; https://github.com/DigitalPebble/storm-crawler; stormcrawler@digitalpebble.com)
BlogBridge 2.13 (http://www.blogbridge.com/)
BlogBridge 6.7 (http://www.blogbridge.com/) 1.6.0_18
BlogPulse (ISSpider-3.0)
BlogRangerCrawler/1.0
BlogSearch/2 +http://www.icerocket.com/
Bloglines/3.0-rho (http://www.bloglines.com; 3 subscribers)
Bloglines/3.1 (http://www.bloglines.com)
Bloglovin/1.0 (http://www.bloglovin.com; 0 subscribers)
Bloglovin/1.0 (http://www.bloglovin.com; 1 subscribers)
BlogramCrawler/1.0.1(+http://blogram.jp/)
Blogshares Spiders (Renewed V1.65)
Blogtrottr/2.0
BoardReader Blog Indexer(http://boardreader.com)
BoardReader Favicon Fetcher /1.0 info@boardreader.com
BoardReader/1.0 (http://boardreader.com/info/robots.htm)-Machine1
Bot Blocker Crawler 1.0 (btw, IncrediBILL says "HI!")
Bot.AraTurka.com/0.0.1
BotOnParade, http://www.bots-on-para.de/bot.html
BrainbruBot/1.0 (+http://www.brainbru.com/)
BrandONbot (http://brandonmedia.net)
BravoBrian SpiderEngine MarcoPolo
Brekiri crawler/1.0
BrightCrawler (http://www.brightcloud.com/brightcrawler.asp)
Brodie/1.0 (+http://www.15miles.com/)
Browsershots
BublupBot (+https://www.bublup.com/bublup-bot.html)
Buck/2.2; (+https://app.hypefactors.com/media-monitoring/about.html)
BuckyOHare/1.3 (Googlebot/2.1; +https://hypefactors.com/webcrawler)
BuiBui-Bot/1.0 (email: buibui[at]dadapro[dot]com)
BuildCMS crawler (http://www.buildcms.com/crawler)
BurstFind Crawler 1.1 - www.burstfind.com
BurstFindCrawler/1.1 (crawler.burstfind.com; http://crawler.burstfind.com; crawler@burstfind.com)
Buzzbot/1.0 (Buzzbot; http://www.buzzstream.com; buzzbot@buzzstream.com)
ByWebSite Search Engine Spider
ByWebSite-Search-Spider,gzip(gfe),gzip(gfe)
CATExplorador/1.0beta (sistemes at domini dot cat; http://domini.cat/catexplorador.html)
CB/Nutch-1.7
CC Metadata Scaper http://wiki.creativecommons.org/Metadata_Scraper
CCBot/1.0 (+http://www.commoncrawl.org/bot.html)
CCBot/2.0
CCBot/2.0 (http://commoncrawl.org/faq/)
CCBot/2.0 (https://commoncrawl.org/faq/)
CCResearchBot/1.0 commoncrawl.org/research//Nutch-1.7-SNAPSHOT
CDR/1.7.1 Simulator/0.7(+http://timewe.net) Profile/MIDP-1.0 Configuration/CLDC-1.0
CERT.at-Statistics-Survey/1.0 (http://www.cert.at/about/consec/content.html)
CFG_SPIDER_USER_AGENT
CISPA Vulnerability Notification (https://notify.mmci.uni-saarland.de)
CISPA Web Analyser (https://notify.cispa.saarland)
CJNetworkQuality; http://www.cj.com/networkquality
CMS crawler ( http://buytaert.net/crawler/)
COMODO DCV
COMODO SSL Checker
COMODOSpider/Nutch-1.2
COMODOspider/Nutch-1.0
CRAZYWEBCRAWLER 0.9.0, http://www.crazywebcrawler.com
CRAZYWEBCRAWLER 0.9.10, http://www.crazywebcrawler.com
CRAZYWEBCRAWLER 0.9.2, http://www.crazywebcrawler.com
CRAZYWEBCRAWLER 0.9.4, http://www.crazywebcrawler.com
CRIM Crawler/Nutch-2.3 (Crawler du Centre de Recherche Informatique de Montr\xc3\xa9al (CRIM))
CSE HTML Validator Online (http://www.OnlineWebCheck.com) via 108.62.195.142
CSS Certificate Spider (http://www.css-security.com/certificatespider/)
CSimpleSpider Robot
CSmartCrawler (email: abuse[at]adsadv[dot]com)
Caliperbot/1.0 (+http://www.conductor.com/caliperbot)
CamontSpider/1.0 +http://epweb2.ph.bham.ac.uk/user/slater/camont/info.html
CamontSpider/1.0 http://epweb2.ph.bham.ac.uk/user/slater/camont/info.html
CapsuleChecker (http://www.capsulink.com/)
Castabot/0.1 (+http://topixtream.com/)
Castro 2, Episode Duration Lookup
CatchBot/1.0; +http://www.catchbot.com
CatchBot/2.0; +http://www.catchbot.com
CatchBot/3.0; +http://www.catchbot.com
CatchBot/5.0; +http://www.catchbot.com
CazoodleBot/0.1 (CazoodleBot Crawler; http://www.cazoodle.com; mqbot@cazoodle.com)
CazoodleBot/CazoodleBot-0.1 (CazoodleBot Crawler; http://www.cazoodle.com/cazoodlebot; cazoodlebot@cazoodle.com)
CheckHost (http://check-host.net/)
CheckMarkNetwork/1.0 (+http://www.checkmarknetwork.com/spider.html)
CheckSite Verification Agent (+http://www.checksite.us)
Checklinks/1.3 (pywikipedia robot; http://toolserver.org/~dispenser/view/Checklinks)
Choosito/1.0 +http://www.choosito.com/
Chrome/49.0.2623.87 (compatible; Wappalyzer; +https://github.com/AliasIO/Wappalyzer)
CipaCrawler/3.0 (info@domaincrawler.com; http://www.domaincrawler.com/www.example.com)
CirrusExplorer/1.1 (http://www.cireu.com/explorer.php)
Citoid (Wikimedia tool; learn more at https://www.mediawiki.org/wiki/Citoid)
Cityreview Robot (+http://www.cityreview.org/crawler/)
Clickagy Intelligence Bot v2
Cliqz Bot (+http://www.cliqz.com)
Cliqzbot
Cliqzbot/0.1 (+http://cliqz.com +cliqzbot@cliqz.com)
Cliqzbot/0.1 (+http://cliqz.com/company/cliqzbot)
CoinCornerBot/1.1 ( https://www.coincorner.com/BitcoinBot)
CommaFeed/1.0 (http://www.commafeed.com)
CommaFeed/2.3.0-SNAPSHOT (https://www.commafeed.com)
CommaFeed/2.4.0-SNAPSHOT (https://github.com/Athou/commafeed)
Comodo HTTP(S) Crawler - http://www.instantssl.com/crawler
Comodo SSL Checker
Comodo Spider 1.1
Comodo Spider 1.2
Comodo-Certificates-Spider
Comodo-Webinspector-Crawler 2.1
Comodo-Webinspector-Crawler 2.2.0, http://www.comodorobot.com
Comodo-Webinspector-Crawler 2.2.1, http://www.comodorobot.com
Comodo-Webinspector-Crawler 2.2.2, http://www.comodorobot.com
Companybook-Crawler (+https://www.companybooknetworking.com/)
Computer_and_Automation_Research_Institute_Crawler (crawler@info.ilab.sztaki.hu)
Contacts Crawler (+http://www.scrapinghub.com)
Content Crawler
ContextAd Bot 1.0
ConveraCrawler/0.9c ( http://www.authoritativeweb.com/crawl)
ConveraCrawler/0.9d (+http://www.authoritativeweb.com/crawl)
ConveraCrawler/0.9e (+http://ews.converasearch.com/crawl.htm)
ConveraMultiMediaCrawler/0.1 ( http://www.authoritativeweb.com/crawl)
CopperEgg/RevealUptime/
CopperEgg/RevealUptime/AmsterdamHolland
CopperEgg/RevealUptime/AmsterdamNL(softlayer)
CopperEgg/RevealUptime/AtlantaGA(linode)
CopperEgg/RevealUptime/AtlantaGAUSA
CopperEgg/RevealUptime/DallasTX(linode)
CopperEgg/RevealUptime/DallasTXUSA
CopperEgg/RevealUptime/DublinIE(aws)
CopperEgg/RevealUptime/DublinIEUSA
CopperEgg/RevealUptime/FrankfurtEU
CopperEgg/RevealUptime/FrankfurtGermany
CopperEgg/RevealUptime/FremontCA(linode)
CopperEgg/RevealUptime/FremontCAUSA
CopperEgg/RevealUptime/LondonUK
CopperEgg/RevealUptime/LondonUK(linode)
CopperEgg/RevealUptime/N.California(aws)
CopperEgg/RevealUptime/N.Virginia(aws)
CopperEgg/RevealUptime/N.VirginiaUSA
CopperEgg/RevealUptime/NewarkNJ(linode)
CopperEgg/RevealUptime/NewarkNJUSA
CopperEgg/RevealUptime/Oregon(aws)
CopperEgg/RevealUptime/OregonUSA
CopperEgg/RevealUptime/SaoPauloBR(aws)
CopperEgg/RevealUptime/SaoPauloBrazil
CopperEgg/RevealUptime/Singapore(aws)
CopperEgg/RevealUptime/SydneyAU(aws)
CopperEgg/RevealUptime/SydneyAustralia
CopperEgg/RevealUptime/TokyoJP
CopperEgg/RevealUptime/TokyoJP(aws)
CopperEgg/RevealUptime/TokyoJP(linode)
CopperEgg/RevealUptime/TokyoJapan
CorporateNewsSearchEngine/Nutch-1.7 (http://pibs.co/news-search-engine)
CorpusCrawler 2.0.0 (http://corpora.fi.muni.cz/crawler/)
CorpusCrawler 2.0.10 (http://corpora.fi.muni.cz/crawler/)
CorpusCrawler 2.0.12 (http://corpora.fi.muni.cz/crawler/)
CorpusCrawler 2.0.13 (http://corpora.fi.muni.cz/crawler/)
CorpusCrawler 2.0.14 (http://corpora.fi.muni.cz/crawler/)
CorpusCrawler 2.0.15 (http://corpora.fi.muni.cz/crawler/)
CorpusCrawler 2.0.17 (http://corpora.fi.muni.cz/crawler/);Project:CzCorpus
CorpusCrawler 2.0.19 (http://corpora.fi.muni.cz/crawler/);Project:CzCorpus
CorpusCrawler 2.0.20 (http://corpora.fi.muni.cz/crawler/);Project:CzCorpus
CorpusCrawler 2.0.21 (http://corpora.fi.muni.cz/crawler/);Project:CzCorpus
CorpusCrawler 2.0.22 (http://corpora.fi.muni.cz/crawler/);Project:CzCorpus
CorpusCrawler 2.0.24 (http://corpora.fi.muni.cz/crawler/);Project:CzCorpus
CorpusCrawler 2.0.25 (http://corpora.fi.muni.cz/crawler/);Project:CzCorpus
CorpusCrawler 2.0.8 (http://corpora.fi.muni.cz/crawler/)
CorpusCrawler 2.0.9 (http://corpora.fi.muni.cz/crawler/)
CosmixCrawler/0.1
Covario-IDS/1.0 (Covario; http://www.covario.com/ids; support at covario dot com)
CovarioIDS/1.1 (http://www.covario.com/ids; support at covario dot com)
CowCrawler/CowCrawler-dev (+http://beta.cow.com),gzip(gfe),gzip(gfe)
CrawlConvera0.1 (CrawlConvera@yahoo.com)
CrawlFire - you can disable this Robot: http://pastebin.de/25277
CrawlWave/1.2 (crawlwave[at]circular.gr http://www.spiderwave.aueb.gr/
Crawler powered by contentDetection (www.mindup.de)
CrawlerBoy Pinpoint.com
Crawllybot/0.1/0.1 (Crawllybot/0.1; http://www.crawlly.com; crawler@crawlly.com)
Crawlzilla/1.0 (Crawlzilla; http://www.crawlzilla.com; crawler@crawlzilla.com)
CronDroid v0.80a http://www.crondroid.com
Cronless/2.0 (+http://cronless.com)
Crowsnest/0.5 (+http://www.crowsnest.tv/)
CrunchBot/1.0 (+http://www.leadcrunch.com/crunchbot)
Curebot/1.0
Curious George - www.analyticsseo.com
Curious George - www.analyticsseo.com/crawler
Cutbot; 1.5; http://cutbot.net/
CyberPatrol SiteCat Webbot (http://www.cyberpatrol.com/cyberpatrolcrawler.asp)
CydralSpider/3.2 (Cydral Image Search; http://www.cydral.com)
DER-bot (+https://goo.gl/9ZibHG)
DNS-Tools Header-Analyzer
DNSGroup/0.1 (DNS Group Crawler; http://www.dnsgroup.com/; crawler@dnsgroup.com)
DNSPod-reporting(http://www.dnspod.cn/reporting)
DRKSpider - Website link validator - http://www.drk.com.ar/spider/,gzip(gfe),gzip(gfe),gzip(gfe)
DWDS-Crawler +http://odo.dwds.de/dwds-crawler.html
Datadog Agent/5.10.1
Datafeedwatch/2.1.x
DataparkSearch/4.35-02122005 ( http://www.dataparksearch.org/)
DataparkSearch/4.36 ( http://www.dataparksearch.org/)
DataparkSearch/4.37-23012006 ( http://www.dataparksearch.org/)
DataparkSearch/4.40.1 (+http://www.dataparksearch.org/)
DataparkSearch/4.53 (+http://dataparksearch.org/bot)
DealGates Bot/1.1 by Luc Michalski (http://spider.dealgates.com/bot.html)
DemandbasePublisherAnalyzer/0.1 (http://www.demandbase.com; info@demandbase.com)
Denodo IECrawler/4.5,gzip(gfe),gzip(gfe)
Dex Social Bot
DialogSearch.com Bot 1.0;http://dialogsearch.com/webmasters
DialogSearch.com Bot 1.4;http://dialogsearch.com/webmasters
Diffbot/0.1
Digg Deeper/v1 (http://digg.com/about)
Digg Feed Fetcher 1.0 (Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_1) AppleWebKit/534.48.3 (KHTML, like Gecko) Version/5.1 Safari/534.48.3)
Digincore crawler bot. See https://www.digincore.com/crawler.html for rules and instructions.
DiscoverEd/Nutch-1.7 (OER search crawler; http://wiki.creativecommons.org/DiscoverEd; webmaster@creativecommons.org)
Disqus/1.0
Dlvr.it/1.0 (http://dlvr.it/)
DoCoMo//2.0 N905i(c100;TB;W24H16) (compatible; Googlebot-Mobile/2.1; +http://www.google.com/bot.html)
DoCoMo/2.0 D903i(c100;TB;W28H20) (compatible; Hatena-Mobile-Gateway/1.2; +http://mgw.hatena.ne.jp/help)
DoCoMo/2.0 N902iS(c100;TB;W24H12)(compatible; moba-crawler; http://crawler.dena.jp/)
DoCoMo/2.0 N905i(c100;TB;W24H16) (compatible; Googlebot-Mobile/2.1; +http://www.google.com/bot.html)
DoCoMo/2.0 P05A(c100;TB;W24H15) (compatible; BaiduMobaider/1.0; http://www.baidu.jp/spider/)
DoCoMo/2.0 P900i(c100;TB;W24H11) (compatible; ichiro/mobile goo; +http://help.goo.ne.jp/help/article/1142/)
DoCoMo/2.0 P900i(c100;TB;W24H11) (compatible; ichiro/mobile goo; +http://search.goo.ne.jp/option/use/sub4/sub4-1/)
DoCoMo/2.0 P900i(c100;TB;W24H11) (compatible; ichiro/mobile goo;+http://search.goo.ne.jp/option/use/sub4/sub4-1/)
DoCoMo/2.0 P900i(c100;TB;W24H11)(compatible; ichiro/mobile goo; http://help.goo.ne.jp/door/crawler.html)
DoCoMo/2.0 P900i(c100;TB;W24H11)(compatible; ichiro/mobile goo;+http://help.goo.ne.jp/door/crawler.html)
DoCoMo/2.0 P901i(c100;TB;W24H11) (compatible; ichiro/mobile goo; +http://help.goo.ne.jp/door/crawler.html)
DoCoMo/2.0 P904i( m65bot/0.1; c; http://m65.jp/bot.html )
DoCoMo/2.0 SH905i(c100;TB;W24H16) (compatible; Mediapartners-Google/2.1; +http://www.google.com/bot.html)
DocWeb Link Crawler (http://doc.php.net)
Domain Re-Animator Bot (http://domainreanimator.com) - support@domainreanimator.com
DomainCrawler/2.0 (info@domaincrawler.com; http://www.domaincrawler.com/bot
DomainStatsBot/1.0 (http://domainstats.io/our-bot)
DomainsDB.net MetaCrawler v.0.9.7b (http://domainsdb.net/)
DomainsDB.net MetaCrawler v.0.9.7c (http://domainsdb.net/)
Domnutch-Bot/Nutch-1.0 (Domnutch; http://www.Nutch.de/)
DotSpotsBot/0.2 (crawler; support at dotspots.com)
DoubleVerify Crawler
DowntimeDetector/1.0 (+http://downforeveryoneorjustme.com)
Drupal (+http://drupal.org/)
DuckDuckBot/1.0; (+http://duckduckgo.com/duckduckbot.html)
DuckDuckBot/1.1; (+http://duckduckgo.com/duckduckbot.html)
DwnldBot (+http://dwnld.me)
EDI/1.6.0 (Edacious & Intelligent Web Crawler)
EDS EXIF/1.0 (https://www.exif-search.com)
ELB-HealthChecker/1.0
EMail Exractor
ERACrawler/1.0
EZID (EZID link checker; https://ezid.cdlib.org/)
Easy-Thumb (https://www.easy-thumb.net/)
EasyBib AutoCite (http://autocite-info.citation-api.com/)
EasyBib AutoCite (http://content.easybib.com/autocite/)
EasyCron/1.0 (https://www.easycron.com/)
EcoGrader Crawler: Beta
EdisterBot (http://www.edister.com/bot.html)
Elmer, the Thinglink ImageBot (http://www.thinglink.com/help/ThinglinkImageBot)
Email%20Extractor%20Lite/26 CFNetwork/720.3.13 Darwin/14.3.0 (x86_64)
Email%20Extractor/1 CFNetwork/760.6.3 Darwin/15.6.0 (x86_64)
Email%20Extractor/29 CFNetwork/897.15 Darwin/17.5.0 (x86_64)
EmailSiphon
Embedly +support@embed.ly
EmeraldShield.com WebBot
EmeraldShield.com WebBot (http://www.emeraldshield.com/webbot.aspx)
EnaBot/1.2 (http://www.enaball.com/crawler.html)
Enterprise_Search/1.0 110 (http://www.innerprise.net/es-spider.asp)
EricssonR320/R1A (Fast Wireless Crawler)
EsperanzaBot(+http://www.esperanza.to/bot/)
EuripBot/0.5 (+http://www.eurip.com) PreCheck
EuripBot/1.1 (+http://www.eurip.com) GetRobots
EuripBot/2.0 (+http://www.eurip.com) GetRobots
Eurobot/1.1 (http://eurobot.ayell.eu)
Eurobot/1.2 (http://eurobot.ayell.eu)
EventMachine HttpClient
EverbeeCrawler
Everwall Link Expander ( http://everwall.com )
ExB Language Crawler 2.1.1 (+http://www.exb.de/crawler)
ExB Language Crawler 2.1.2 (+http://www.exb.de/crawler)
ExB Language Crawler 2.1.5 (+http://www.exb.de/crawler)
ExactSeek Crawler (http://www.exactseek.com/)/Nutch-1.4
ExactSeek Crawler (nutch 1.4)/Nutch-1.4 (ExactSeek Crawler; http://www.exactseek.com)
ExactSeek Crawler/0.1
ExactSeekCrawler/1.0
ExactSeek_Spider
Exalead Cloudview Crawler,gzip(gfe),gzip(gfe),gzip(gfe)
Expanse indexes the network perimeters of our customers. If you have any questions or concerns, please reach out to: scaninfo@expanseinc.com
Expanse, a Palo Alto Networks company, searches across the global IPv4 space multiple times per day to identify customers&#39; presences on the Internet. If you would like to be excluded from our scans, please send IP addresses/domains to: scaninfo@paloaltonetworks.com
Experibot-v2 http://goo.gl/ZAr8wX
Experibot-v3 http://goo.gl/ZAr8wX
Experibot_v1
Experibot_v1 (https://dl.dropboxusercontent.com/u/8024465/site/Info.html)
Experibot_v1 [bit.ly/2bLMJdw]
Experibot_v1 [goo.gl/n6zrAf]
ExtractorPro
FAST Enteprise Crawler/6 (www dot fastsearch dot com)
FAST Enterprise Crawler 6 / Scirus scirus-crawler@fast.no; http://www.scirus.com/srsapp/contactus/
FAST Enterprise Crawler 6 used by Schibsted (webcrawl@schibstedsok.no)
FAST Enterprise Crawler/5.3.4 (crawler@fast.no)
FAST EnterpriseCrawler 6
FAST MetaWeb Crawler (helpdesk at fastsearch dot com)
FAST-WebCrawler/2.2.11 (crawler@fast.no; http://www.fast.no/faq/faqfastwebsearch/faqfastwebcrawler.html)
FAST-WebCrawler/3.6/FirstPage (atw-crawler at fast dot no;http://fast.no/support/crawler.asp)
FAST-WebCrawler/3.7 (atw-crawler at fast dot no; http://fast.no/support/crawler.asp)
FAST-WebCrawler/3.7/FirstPage (atw-crawler at fast dot no;http://fast.no/support/crawler.asp)
FAST-WebCrawler/3.8
FAST-WebCrawler/3.8 (atw-crawler at fast dot no; http://fast.no/support/crawler.asp)
FAST-mSEARCH Crawler 0.1 (bergum@fast.no)
FDM 3.x
FFC Trap Door Spider
FFbot/1.0; +http://feedfury.com
FTRF: Friendly robot/1.3
Facebot/1.0
Factbot 1.09 (see http://www.factbites.com/webmasters.php)
Fast PartnerSite Crawler
Fastladder FeedFetcher/0.01 (http://fastladder.com/; 27 subscribers)
Favicon downloader (+https://favico.be/)
Favicon downloader (+https://favico.be/bot.html)
Favicon downloader (http://favicon.netk6.com/)
FaviconArchiver/1.0 (+http://moblur.org/workshop/favicon_archive/)
Feed Seeker Bot (RSS Feed Seeker http://www.MyNewFavoriteThing.com/fsb.php)
Feed Wrangler/1.0 (3 subscribers; feed-id=248559; http://feedwrangler.net; Allow like Gecko)
FeedBucket/1.0 (+http://www.feedbucket.com)
FeedBunch/1.0 (+http://www.feedbunch.com)
FeedBurner/1.0 (http://www.FeedBurner.com)
FeedCatBot/3.0 (+http://www.feedcat.net/)
FeedChecker-Zocle/1.0 (+https://zocle.com/zoclechecker)
FeedDemon/4.5 (http://www.feeddemon.com/; Microsoft Windows)
FeedFetcher-Google; (+http://www.google.com/feedfetcher.html)
FeedValidator/1.3
FeedZcollector v1.x (Platinum) http://www.feeds4all.com/feedzcollector
Feedbin
Feedbin - 9 subscribers
Feedfetcher-Google; (+http://www.google.com/feedfetcher.html; 1 subscribers; feed-id=728742641706423)
Feedfetcher-Google; (+http://www.google.com/feedfetcher.html; 19 subscribers; feed-id=13965549748850348809)
Feedfetcher-Google; (+http://www.google.com/feedfetcher.html; 2 subscribers; feed-id=17860707833818568603)
Feedfetcher-Google; (+http://www.google.com/feedfetcher.html; 375 subscribers; feed-id=15381863289700640853)
Feedfetcher-Nuesbyte; (+http://www.nuesbyte.com)
Feedjit Favicon Crawler 1.0
Feedly/1.0 (+http://www.feedly.com/fetcher.html; like FeedFetcher-Google)
FeedlyApp/1.0 (http://www.feedly.com)
FeedlyBot/1.0 (http://feedly.com)
Feedshow/x.0 (http://www.feedshow.com; 1 subscriber)
FeedshowOnline (http://www.feedshow.com)
Feedspot http://www.feedspot.com
Fetch/2.0a (CMS Detection/Web/SEO analysis tool, see http://guess.scritch.org)
Fever/1.38 (Feed Parser; http://feedafever.com; Allow like Gecko)
Finderlein Research Crawler 1.0
Findexa Crawler (http://www.findexa.no/gulesider/article26548.ece)
Flamingo_SearchEngine (+http://www.flamingosearch.com/bot)
FlaxCrawler/1.0,gzip(gfe),gzip(gfe)
FlightDeckReportsBot/2.0 (http://www.flightdeckreports.com/pages/bot)
FollowSite Bot ( http://www.followsite.com/bot.html )
FreeWebMonitoring SiteChecker/0.1 (+http://www.freewebmonitoring.com)
FreeWebMonitoring SiteChecker/0.2 (+http://www.freewebmonitoring.com/bot.html)
FreeWebMonitoring SiteChecker/0.2 (+https://www.freewebmonitoring.com/bot.html)
FreshRSS/1.11.2 (Linux; https://freshrss.org) like Googlebot
FreshRSS/1.12.0 (Linux; https://freshrss.org)
FreshRSS/1.13.0 (Linux; https://freshrss.org)
Friendica 'Lily of the valley' 3.4.0-1182; http://frndc.saschaschroeder.eu
Friendica 'The Tazmans Flax-lily' 2019.01-1293; https://hoyer.xyz
Fujiko Spider (fujiko@inelegant.org)
FyberSpider
FyberSpider/1.3 (http://www.fybersearch.com/fyberspider.php)
Fyrebot/1.0
G2 Web Services/1.0 (built with StormCrawler Archetype 1.8; https://www.g2webservices.com/; developers@g2llc.com)
GAChecker (+http://www.gachecker.com)
GC3pro+dir SEO Tools - Vers. 3.00b - For more informations: http://chkme.com/
GG PeekBot 2.0 ( http://gg.pl/ http://info.gadu-gadu.pl/praca )
GIDBot/2.0 (+http://www.gidnetwork.com/tools/gzip-test.php)
GIDBot/3.0 (+http://www.gidnetwork.com/tools/gzip-test.php)
GOFORITBOT ( http://www.goforit.com/about/ )
GSiteCrawler/v1.12 rev. 260 (http://gsitecrawler.com/)
Gaisbot/3.0+(robot06@gais.cs.ccu.edu.tw;+http://gais.cs.ccu.edu.tw/robot.php)
GalaxyBot/1.0 (http://www.galaxy.com/galaxybot.html)
GarlikCrawler/1.1 (http://garlik.com/, crawler@garik.com)
GarlikCrawler/1.1 (http://garlik.com/, crawler@garlik.com)
GarlikCrawler/1.2 (http://garlik.com/, crawler@garlik.com)
GeekTools Crawler - http://domains.geek-tools.org
GematchCrawler/2.1 (http://www.gematch.com/crawler.html)
Genderanalyzer/1.0
GentleSource Short URL Checker ( http://www.gentlesource.com/ )
GermCrawler
GetFoundBot (+http://www.getfound.cz/getfoundbot/)
GetIntent Crawler (http://getintent.com/bot.html)
GetProxi.es-bot/1.1 (http://getproxi.es/spiderinfo/)
GetintentCrawler getintent.com
GigablastOpenSource/1
GigablastOpenSource/1.0
Gigabot/1.0
Gigabot/2.0
Gigabot/2.0 (http://www.gigablast.com/spider.html)
Gigabot/3.0 (http://www.gigablast.com/spider.html)
GingerCrawler/1.0 (Language Assistant for Dyslexics; www.gingersoftware.com/crawler_agent.htm; support at ginger software dot com)
Girls/2.0 (livedoor Co.,Ltd.; Peachy 2.1; iPhone; RSS Version 2.0; +http://girls.livedoor.com/)
GloomarBot/1.1 (https://www.gloomar.com/bot)
Gnam Gnam Spider
Go-http-client
Go-http-client/1.1
Go-http-client/2.0
GoGuidesBot/0.0.1 (GoGuides Indexing Spider; http://www.goguides.org/spider.html)
GoScraper
GoSquared-Status-Checker/0.2
Google (+https://developers.google.com/+/web/snippet/)
Google favicon
Google-AdWords-Express
Google-Ads-Overview Mozilla/5.0 (Linux; U; Android 2.3.4; generic) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.118 Version/4.0 Mobile Safari/537.36
Google-Ads-Overview Mozilla/5.0 (Linux; U; Android 2.3.4; generic) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Mobile Safari/537.36
Google-Ads-Overview Mozilla/5.0 (Linux; U; Android 6.0.1; generic) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.118 Version/4.0 Mobile Safari/537.36
Google-Ads-Overview Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.118 Safari/537.36
Google-Adwords-Instant (+http://www.google.com/adsbot.html)
Google-Calendar-Importer
Google-Certificates-Bridge
Google-HTTP-Java-Client/1.13.1-beta (gzip)
Google-HTTP-Java-Client/1.17.0-rc (gzip)
Google-HTTP-Java-Client/1.9.0-alpha
Google-SMTP-STS
Google-Sitemaps/1.0
Google-Structured-Data-Testing-Tool +https://search.google.com/structured-data/testing-tool)
Google-Transparency-Report
Google-YouTubeSample/1.0 Google-HTTP-Java-Client/1.7.0-beta (gzip) Google-HTTP-Java-Client/1.7.0-beta (gzip)
Google-speakr
Google-speakr,gzip(gfe)
GoogleCloudMonitoring
GoogleProducer; (+http://goo.gl/7y4SX)
GoogleStackdriverMonitoring-UptimeChecks
Googlebot (gocrawl v0.4)
Googlebot deepcrawl
Googlebot-Image/1.0
Googlebot-Mobile (compatible; Googlebot-Mobile/2.1; +http://www.google.com/bot.html)
Googlebot-News
Googlebot-News (2.3.3, ruby 1.9.3 (2013-11-22))
Googlebot-Video/1.0
Googlebot-richsnippets
Googlebot/2.1 (+http://www.google.com/bot.html)
Googlebot/2.1 (http://www.googlebot.com/bot.html)
Googlebot/Nutch-1.7
GozaikBot (www.gozaik.com;webmaster@gozaik.com;www.gozaik.com/gozaikbot.html)
Grabber -  https://sites.google.com/3ziele.de/grabber/
Grahambot/0.1 (+http://www.sunaga-lab.com/graham-bot)
Grammarly/1.0 (http://www.grammarly.com)
Gregarius/0.5.2 ( http://devlog.gregarius.net/docs/ua)
Gregarius/0.5.2 (+http://devlog.gregarius.net/docs/ua)
GrowerIdeas Crawler/GrowerIdeas-nutch-1.6 (Crawls URLs for indexing content for our new search startup which aims to provide simple and smart search across curated content. For more info please contact helloworld@growerideas.com. If you think
GuzzleHttp/6.1.0 curl/7.38.0 PHP/7.0.0RC7
GuzzleHttp/6.2.1 PHP/7.0.15-0ubuntu0.16.04.4
Gwene/1.0 (The gwene.org rss-to-news gateway) Googlebot
HPI FeedCrawler/0.1 (+http://www.hpi.uni-potsdam.de/meinel/bross/feedcrawler)
HRCrawler/2.0 (XF86; MacOS x86_64) AppleWebKit/537.31 (KHTML, like Gecko)
HTTP Banner Detection (https://security.ipip.net)
HTTP-Header-Abfrage/1.0 (http://www.internalscripts.de/werkzeuge/http-header-abfrage.php)
HTTPClient/1.0 (2.4.0, ruby 1.9.3 (2013-06-27))
HTTPClient/1.0 (2.8.0, ruby 2.0.0 (2014-05-08))
HTTPMon/1.0b (http://www.httpmon.com)
HTTP_Request2/2.1.1 (http://pear.php.net/package/http_request2) PHP/5.3.10-1ubuntu3.6
HTTP_Request2/2.2.0 (http://pear.php.net/package/http_request2) PHP/5.3.29
HTTP_Request2/2.2.1 (http://pear.php.net/package/http_request2) PHP/5.3.29
HTTP_Request2/2.2.1 (http://pear.php.net/package/http_request2) PHP/5.5.33
HTTP_Request2/2.2.1 (http://pear.php.net/package/http_request2) PHP/5.5.35
HTTP_Request2/2.3.0 (http://pear.php.net/package/http_request2) PHP/5.3.29
HTTP_Request2/@package_version@ (http://pear.php.net/package/http_request2) PHP/5.5.9-1ubuntu4.14
HappyFunBot/1.1 ( http://www.happyfunsearch.com/bot.html)
Hardenize (https://www.hardenize.com)
Hatena Antenna/0.3
Hatena Antenna/0.5 (http://a.hatena.ne.jp/help)
Hatena Diary RSS Module (http://d.hatena.ne.jp/help#rssmodule)
Hatena Pagetitle Agent/1.0
Hatena-Favicon/2 (http://www.hatena.ne.jp/faq/)
Hatena-Favicon2 (http://www.hatena.ne.jp/faq/)
Hatena::Bookmark/2.00 (Hatena::Bookmark; master;)
Hatena::Crawler/0.01
Hatena::Fetcher/0.01 (master) Furl/3.13
Hatena::Russia::Crawler/0.01
Hatena::Scissors/0.01
HatenaBookmark/4.0 (Hatena::Bookmark; Analyzer)
HatenaScreenshot
HatenaScreenshot/1.0 (checker)
HeartRails Robot/0.1 (http://www.heartrails.com)
HetrixTools.COM Uptime Monitoring Bot. https://hetrixtools.com/uptime-monitoring-bot.html
Heurekabot-Feed/1.0 (+http://sluzby.heureka.cz/napoveda/heurekabot/)
Heurekabot-Feed/1.0 (+https://sluzby.heureka.cz/napoveda/heurekabot/)
Heurekabot-ImageFullText/1.0 (+http://sluzby.heureka.cz/napoveda/heurekabot/)
Heurekabot-ImageFullText/1.0 (+https://sluzby.heureka.cz/napoveda/heurekabot/)
HggH PhantomJS Screenshoter
HiddenMarket-1.0-beta (www.hiddenmarket.net/crawler.php)
HolmesBot (http://holmes.ge)
Hong Kong Education City/1.0 (built with Hong Kong Education City 1.10; http://www.hkedcity.net/; search@hkedcity.net)
HooWWWer/2.1.0 ( http://cosco.hiit.fi/search/hoowwwer/ | mailto:crawler-info<at>hiit.fi)
HttpSpider/0.91
HuaweiSymantecSpider/1.0 DSE-support@huaweisymantec.com (compatible; MSIE 7.0; Windows NT 5.1; Trident/4.0; .NET CLR 2.0.50727; .NET CLR 3.0.4506.2152; .NET CLR ; http://www.huaweisymantec.com/cn/IRL/spider)
HuaweiSymantecSpider/1.0+DSE-support@huaweisymantec.com+(compatible; MSIE 7.0; Windows NT 5.1; Trident/4.0; .NET CLR 2.0.50727; .NET CLR 3.0.4506.2152; .NET CLR ; http://www.huaweisymantec.com/cn/IRL/spider)
HuaweiSymantecSpider/1.0+DSE-support@huaweisymantec.com+(compatible; MSIE 7.0; Windows NT 5.1; Trident/4.0; .NET CLR 2.0.50727; .NET CLR 3.0.4506.2152; .NET CLR ; http://www.huaweisymantec.com/en/IRL/spider)
HuaweiWebCatBot/6.0) (To acquire the allowed html pages as reliable information of URL categorization in the automatic process for Huawei Web Categorization.; https://isecurity.huawei.com/; sec at huawei dot com)
Huaweisymantecspider (compatible; MSIE 8.0; DSE-support@huaweisymantec.com)
HubPages V0.2.2 (http://hubpages.com/help/crawlingpolicy)
HubSpot Connect 1.0 (http://dev.hubspot.com/)
HubSpot Connect 2.0 (http://dev.hubspot.com/)
HubSpot Connect 2.0 (http://dev.hubspot.com/) - BizOpsCompanies-Tq2-BizCoDomainValidationAudit
HubSpot Content Strategy link check
HubSpot Crawler 1.0 http://www.hubspot.com/
HubSpot Links Crawler 1.0 http://www.hubspot.com/
HubSpot Links Crawler 2.0 http://www.hubspot.com/
HubSpot Marketing Grader
HubSpot Marketing Grader, HubSpot Marketing Grader
HubSpot Page Screenshot Service
HubSpot Website Grader (web-crawlers@hubspot.com)
HubSpot inbound link reporting check
HubSpot-Link-Resolver
HybridBot (hybrid.ru/about. If our bot caused problems please contact us. Contact email: m.lyashkov@targetix.net)
IAS crawler (ias_crawler; http://integralads.com/site-indexing-policy/)
IAS crawler (page scorer; http://integralads.com/site-indexing-policy/)
IC OpenGraph Crawler 4.5 (proprietary)
ICC-Crawler(Mozilla-compatible; ; http://kc.nict.go.jp/project1/crawl.html)
ICC-Crawler/2.0 (Mozilla-compatible; ; http://kc.nict.go.jp/project1/crawl.html)
ICC-Crawler/2.0 (Mozilla-compatible; ; http://ucri.nict.go.jp/en/icccrawler.html)
ICC-Crawler/2.0 (Mozilla-compatible; ; http://www.nict.go.jp/en/ucri/plan/crawl.html)
ICC-Crawler/2.0 (Mozilla-compatible; ; http://www.nict.go.jp/en/univ-com/plan/crawl.html)
IDG/IT (http://spaziodati.eu/)
INGRID/2.0 (http://spsearch.ilse.nl/; Startpagina dochter links spider)
INGRID/3.0 MT (webcrawler@NOSPAMexperimental.net; http://aanmelden.ilse.nl/?aanmeld_mode=webhints)
IOI/2.0 (ISC Open Index crawler; http://index.isc.org/; bot@index.isc.org)
IP-Guide.com Crawler/1.0 (https://ip-guide.com)
IPv4Scan (+http://ipv4scan.com)
IRLbot/2.0 ( http://irl.cs.tamu.edu/crawler)
IRLbot/2.0 (+http://irl.cs.tamu.edu/crawler)
IRLbot/2.0 (compatible; MSIE 6.0; http://irl.cs.tamu.edu/crawler)
IRLbot/3.0 (compatible; MSIE 6.0; http://irl.cs.tamu.edu/crawler)
IRLbot/3.0 (compatible; MSIE 6.0; http://irl.cs.tamu.edu/crawler/)
IS Alpha/zcspider-0.1
IWE Spider v.01 - www.pavka.com.au
IXE Crawler
IZaBEE/IZaBEE-1.01 (Buzzing Abound The Web; https://izabee.com; info at izabee dot com)
IconSurf/2.0 favicon finder (see http://iconsurf.com/robot.html)
Iframely/0.6.0 (+http://iframely.com/;)
Iframely/0.7.1 (+http://iframe.ly/;)
Iframely/0.8.5 (+http://iframely.com/;)
Iframely/0.8.8 (+http://iframely.com/;)
Iframely/0.8.8 (+https://iframely.com/;)
Iframely/0.9.8 (+http://iframely.com/;)
Iframely/0.9.8 (+https://iframely.com/;)
Iframely/1.0.0 (+https://iframely.com/;)
IlTrovatore-Setaccio ( http://www.iltrovatore.it)
IlTrovatore-Setaccio/1.2 (http://www.iltrovatore.it/aiuto/faq.html)
IlTrovatore/1.2 (IlTrovatore; http://www.iltrovatore.it/bot.html; bot@iltrovatore.it)
Imperia-LinkSpider,gzip(gfe),gzip(gfe)
ImplisenseBot 1.0
ImplisenseBot 1.1
InAGist URL Resolver (http://inagist.com)
Inar_spider2 (tspyyp@tom.com)
Influencebot/0.9; (Automatic classification of websites; http://www.influencebox.com/; info@influencebox.com)
Influencebot/0.9; (Automatic classification of websites; http://www.influencebox.com/; info@influencebox.com)User-Agent: Mozilla/5.0 (X11; Linux i686; rv:9.0) Gecko/20100101 Firefox/9.0
InfoWizards Reciprocal Link System PRO - (http://www.infowizards.com)
Inspingbot/1.0 (+https://www.insping.com/)
InterNaetBoten/0.99 (http://www.xn--internt-bxa.nu)
Interdose AntiSpamBot/2.0 (+http://www.piep.net)
InternetArchive-1.0
InternetArchive/0.8-dev (Nutch; http://lucene.apache.org/nutch/bot.html; nutch-agent@lucene.apache.org)
Irokez.cz monitoring v1.2 - (http://www.irokez.cz, Irokez.cz, crawl)
Is is up? (+http://isitup.org)
Isara-Search/Isara-1.0 (A non-profit web crawler operated by a charity organization.; www.isara.org; webmaster@isara.org)
IssueCrawler
JS-Kit URL Resolver http://js-kit.com/
JUST-CRAWLER( http://www.justsystems.com/jp/tech/crawler/)
JUST-CRAWLER(+http://www.justsystems.com/jp/tech/crawler/)
Jabse.com Crawler v.1.0 www.jabse.com/crawler.php//imagecrawler
Jakarta Commons-HttpClient/3.0
Jambot/0.1.1 (Jambot; http://www.jambot.com/blog; crawler@jambot.com)
Jamie's Spider (http://jamiembrown.com/)
Java/1.17
Java/1.4.1_04
Java/1.6.0
Java/1.6.0_04
Java/1.6.0_22
Java/1.6.0_24
Java/1.6.0_26
Java/1.6.0_31
Java/1.6.0_32
Java/1.6.0_34
Java/1.6.0_37
Java/1.7.0_101
Java/1.7.0_45
Java/1.7.0_51
Java/1.7.0_65
Java/1.7.0_80
Java/1.8.0_05
Java/1.8.0_20
Java/1.8.0_31
Java/1.8.0_40
Java/1.8.0_45
Java/1.8.0_51
Java/1.8.0_60
Java/1.8.0_65
Java/1.8.0_66
Java/1.8.0_73
Java/1.8.0_77
Java/1.8.0_91
Java/1.8.0_92
Jaxified Bot (+http://www.jaxified.com/crawler/)
Jaxified Crawler 1.0a (+http://www.jaxified.com/),gzip(gfe),gzip(gfe)
Jaxified Public RSS Crawler ( http://www.jaxified.com/)
Jayde Crawler. http://www.jayde.com
Jeffrey's Exif Viewer (http://regex.info/exif)
Jersey/2.19 (HttpUrlConnection 1.7.0_85)
Jersey/2.25.1 (Apache HttpClient 4.5)
Jersey/2.25.1 (HttpUrlConnection 1.8.0_141)
JetBrains Omea Reader 1.0.x (http://www.jetbrains.com/omea_reader/)
JetBrains Omea Reader 2.0 Release Candidate 1 (http://www.jetbrains.com/omea_reader/)
JetBrains Omea Reader 2.2 (http://www.jetbrains.com/omea/reader/)
Jetty/9.3.z-SNAPSHOT
Jetty/9.3.z-SNAPSHOT, SMUrlExpander
Jetty/9.3.z-SNAPSHOT, SMViewer
Jigsaw/2.3.0 W3C_CSS_Validator_JFouffa/2.0
Jigsaw/2.3.0 W3C_CSS_Validator_JFouffa/2.0 (See <http://validator.w3.org/services>)
JikeSpider Mozilla/5.0 (compatible; JikeSpider; +http://shoulu.jike.com/spider.html)
JikeSpider; +http://shoulu.jike.com/spider.html
JoBo/1.x (http://www.matuschek.net/jobo.html)
JobFeed discovery script (http://www.jobfeed.com)
JobRoboter/1.1
JobSpider_BA/1.1
JolokiaPwn, https://matmannion.com/jolokiapwn/, mat.mannion@gmail.com
Jomjaibot/1.0 Crawl (+http://www.jomjaibot.com/)
JoobleBot (compatible; http://jooble.org/jooble-bot)
Jugendschutzprogramm-Crawler; Info: http://www.jugendschutzprogramm.de
Jyxobot/1
JyxobotRSS/0.06
K7MLWCBot/1.0 (+http://www.k7computing.com)
KD Bot
KDDI-CA31 UP.Browser/6.2.0.7.3.129 (GUI) MMP/2.0 (compatible; ichiro/mobile goo; +http://help.goo.ne.jp/help/article/1142/)
KDDI-CA31 UP.Browser/6.2.0.7.3.129 (GUI) MMP/2.0 (compatible; ichiro/mobile goo; +http://search.goo.ne.jp/option/use/sub4/sub4-1/)
KDDI-CA31 UP.Browser/6.2.0.7.3.129 (GUI) MMP/2.0 (compatible; ichiro/mobile goo;+http://search.goo.ne.jp/option/use/sub4/sub4-1/)
KDDI-CA34 UP.Browser/6.2.0.10.2.2(GUI)MMP/2.0 (compatible; KDDI-Googlebot-Mobile/2.1; http://www.google.com/bot.html)
KIT webcrawler/0.2.4
KOCMOHABT (+http://kozmonavt.tk/) AppEngine-Google; (+http://code.google.com/appengine; appid: s~web-explorer)
KOCMOHABT (+http://kozmonavt.tk/) Mozilla/5.0 (compatible; Web Explorer 1) AppEngine-Google; (+http://code.google.com/appengine; appid: s~web-explorer)
KOCMOHABT (+http://www.KOCMOHABT.tk/) AppEngine-Google; (+http://code.google.com/appengine; appid: s~web-explorer)
KOCMOHABT (+https://kozmonavt.tk/) Mozilla/5.0 (compatible; Web Explorer 2)
Karneval-Bot (Version: 1.06, powered by www.karnevalsuchmaschine.de +http://www.karnevalsuchmaschine.de/bot.html)
Kaspersky Lab CFR link resolver cfradmins@kaspersky.com
Kemvibot/1.0 (http://kemvi.com)
Kemvibot/1.0 (http://kemvi.com, marco@kemvi.com)
Keyword Extractor 5000; lucas@hyscore.io
KeywordDensityRobot/0.8 (http://www.seocentro.com/tools/search-engines/keyword-density.html)
KeywordDensityRobot/0.9 (http://www.seocentro.com/tools/search-engines/keyword-density.html)
KiNShooboT (compatible; KiNShooboT/1.0.C; +http://www.kinshoo.com/bot.html)
KiwiStatus (NZS.com)/0.2 (NZS.com KiwiStatus Spider, Local Search New Zealand; http://www.nzs.com; bot-at-nzs dot com)
Kml-Google; (+http://code.google.com/apis/kml) gzip
Kotoss Crawler http://hitokoto.kotoss.com,gzip(gfe),gzip(gfe)
Kurzor/1.0 (Kurzor; http://adcenter.hu/docs/en/bot.html; cursor@easymail.hu)
Kyoto-Crawler/2.0 (Mozilla-compatible; kyoto-crawler-contact(at)nlp(dot)kuee(dot)kyoto-u(dot)ac(dot)jp; http://nlp.ist.i.kyoto-u.ac.jp/)
Kyoto-Crawler/n1.0 (Mozilla-compatible; kyoto-crawler-contact@nlp.ist.i.kyoto-u.ac.jp; http://nlp.ist.i.kyoto-u.ac.jp/?crawling)
Kyoto-Tohoku-Crawler/v1 (Mozilla-compatible; kyoto-crawler-contact@nlp.ist.i.kyoto-u.ac.jp; http://nlp.ist.i.kyoto-u.ac.jp/?crawling-kt)
L.webis/0.44 (http://webalgo.iit.cnr.it/index.php?pg=lwebis)
L.webis/0.50 (http://webalgo.iit.cnr.it/index.php?pg=lwebis)
L.webis/0.51 (http://webalgo.iit.cnr.it/index.php?pg=lwebis)
L.webis/0.53 (http://webalgo.iit.cnr.it/index.php?pg=lwebis)
L.webis/0.87 (http://webalgo.iit.cnr.it/index.php?pg=lwebis)
LB-Crawler/1.0
LCC (+http://corpora.informatik.uni-leipzig.de/crawler_faq.html)
LEIA/3.01pr (LEIAcrawler; leia@gseek.com; http://www.gseek.com)
LG-C1500 UP.Browser/6.2.3 (GUI) MMP/1.0 (compatible;YahooSeeker/M1A1-R2D2; http://help.yahoo.com/help/us/ysearch/crawling/crawling-01.html)
LNSpiderguy
LOOQ/0.1 alfa (LOOQ Crawler for european sites; http://looq.eu; root (at) looq dot eu)
LSSRocketCrawler/1.0 LightspeedSystems
LYT.SR v1.5 +http://lyt.sr/ (running on http://lyt.sr)
Landau-Media-Spider/1.0(http://bots.landaumedia.de/bot.html)
LapozzBot/1.4 (+http://robot.lapozz.com)
LarbinWebCrawler (internet@bredband.net)
LargeSmall Crawler
Leikibot/1.0 (+http://www.leiki.com)
LexxeBot/1.0 (lexxebot@lexxe.com)
Liferea/0.x.x (Linux; en_US.UTF-8; http://liferea.sf.net/)
Liferea/1.6.4 (Linux; en_US.UTF-8; http://liferea.sf.net/)
Lijit Crawler (+http://www.lijit.com/robot/crawler)
LinguaBot/v0.001-dev (MultiLinual Sarch Engine v0.001; LinguaSeek; admin at linguaseek dot com)
Linguee Bot (bot@linguee.com)
Linguee Bot (http://www.linguee.com/bot)
Linguee Bot (http://www.linguee.com/bot; bot@linguee.com)
Link Validity Check From: http://www.w3dir.com/cgi-bin (Using: Hot Links SQL by Mrcgiguy.com)
LinkAider (http://linkaider.com/crawler/)
LinkPreview/1.5 (http://linkpreview.net)
LinkStar Research Crawler (http://linkstar.com/),gzip(gfe),gzip(gfe)
LinkStats Bot
LinkWalker/3.0 (http://www.brandprotect.com)
LinkedInBot/1.0 (compatible; Mozilla/5.0; Apache-HttpClient +http://www.linkedin.com)
LinkedInBot/1.0 (compatible; Mozilla/5.0; Apache-HttpClient +http://www.linkedin.com), libot/Nutch-1.9 (http://www.linkedin.com; libot@linkedin.com)
LinkedInBot/1.0 (compatible; Mozilla/5.0; Jakarta Commons-HttpClient/3.1 +http://www.linkedin.com)
LinkedInBot/1.0 (compatible; Mozilla/5.0; Jakarta Commons-HttpClient/3.1 http://www.linkedin.com)
LinkedInBot/1.0 (compatible; Mozilla/5.0; Jakarta Commons-HttpClient/4.3 +http://www.linkedin.com)
Links4US-Crawler, ( http://links4us.com/)
LinksCrawler 0.1beta
LinqiaMetadataDownloaderBot/1.0 (eng@linqia.com)
LinqiaRSSBot/1.0 (eng@linqia.com)
LinqiaScrapeBot/1.0 (eng@linqia.com)
Liquida Spider 1.0 +http://liquida.com/
Liquida.it-Crawler/1.0 ( crawler@liquida.it +http://www.liquida.it )
Lite Bot0316B
LivelapBot/0.2 (http://site.livelap.com/crawler)
Livelapbot/0.1
Llaut/1.0 (http://mnm.uib.es/~gallir/llaut/bot.html)
LoadImpactPageAnalyzer/1.3.0 (Load Impact; http://loadimpact.com/)
LoadImpactRload/3.0.3 (Load Impact; http://loadimpact.com);
LoadImpactRload/3.1.1 (Load Impact; http://loadimpact.com);
LoadImpactRload/3.1.5 (Load Impact; http://loadimpact.com);
LoadImpactRload/3.2.0 (Load Impact; http://loadimpact.com);
Locubot (compatible; Googlebot; msnbot)
Logict IPv6 Crawler/1.0 (http://ipv6search.logict.net),gzip(gfe),gzip(gfe)
LookUpCrawler - lookupcanada.ca [ZSEBOT]
LumpImageSearch/0.1 (+http://lump.co/about/bot)
Lycos_Spider_(modspider)
MB-SiteCrawler
MBCrawler/1.0 (https://monitorbacklinks.com)
MIA DEV/search:robot/0.0.1 (This is the MIA Bot - crawling for mia research project. If you feel unhappy and do not want to be visited by our crawler send an email to spider@neofonie.de; http://spider.neofonie.de; spider@neofonie.de)
MJ12bot/v1.2.0 (http://majestic12.co.uk/bot.php?+)
MLBot (www.metadatalabs.com/mlbot)
MPICrawler/0.1 +http://kermit.news.cs.nyu.edu/crawler.html,gzip(gfe),gzip(gfe)
MQbot http://metaquerier.cs.uiuc.edu/crawler
MSNBot/Nutch-1.5.1
MSR-ISRCCrawler
MSRBOT
MSRBOT (http://research.microsoft.com/research/sv/msrbot)
MSRBOT (http://research.microsoft.com/research/sv/msrbot/)
MTRobot/0.2 (Metrics Tools Analytics Crawler; https://metrics-tools.de/robot.html; crawler@metrics-tools.de)
MXT/Nutch-1.10 (http://t.co/GSRLLKex24; informatique at mixdata dot com)
MXT/Nutch-1.12-SNAPSHOT (http://t.co/GSRLLKex24; informatique at mixdata dot com)
MagpieRSS/0.61 (+http://magpierss.sf.net)
MagpieRSS/0.72 (+http://magpierss.sf.net)
MagpieRSS/0.72 \(+http://magpierss.sf.net\)
MarketBrewBot/2.1 (+http://www.marketbrew.com/)
MauiBot (crawler.feedback+wc@gmail.com)
MaxPoint Bot (+http://www.maxpoint.com)
MaxPointCrawler/Nutch-1.1 (maxpoint.crawler at maxpointinteractive dot com)
MaxPointCrawler/Nutch-1.10 (maxpoint.crawler at maxpointinteractive dot com)
MeMoNewsBot/2.0 (http://www.memonews.com/en/crawler)
MedSpider v0.0.1
Mediapartners-Google
Mediatoolkitbot (complaints@mediatoolkit.com)
Mediatoolkitbot (info@mediatoolkit.com)
Mediumbot-MetaTagFetcher/0.1 (+https://medium.com/)
Mediumbot-MetaTagFetcher/0.3 (+https://medium.com/)
Mediumbot-ProxyFetcher/0.1 (+https://medium.com/)
MeltwaterNews www.meltwater.com
Melvil Favicon/1.0 (Favicon fetcher for Melvil)
MergadoBot
Mergadobot/3.0.2 (+http://mergado.cz)
MergeFlow-PageReader/0.91;+(+http://mergeflow.net/info/pagereader) Mozilla/5.0 (Windows) compatible
MetaCommentBot; http://metacomment.io/about
MetaFeedly/1.0 (http://www.feedly.com)
MetaGeneratorCrawler/1.1 (www.metagenerator.info)
MetaGeneratorCrawler/1.3.12 (www.metagenerator.info)
MetaGeneratorCrawler/1.3.14 (www.metagenerator.info)
MetaGeneratorCrawler/1.3.2 (www.metagenerator.info)
MetaGeneratorCrawler/1.3.3 (www.metagenerator.info)
MetaGeneratorCrawler/1.3.4 (www.metagenerator.info)
MetaGeneratorCrawler/1.3.8 (www.metagenerator.info)
MetaGeneratorCrawler/1.3.9 (www.metagenerator.info)
MetaHeadersBot (+http://www.metaheaders.com/bot.html)
MetaInspector/2.2.1 (+https://github.com/jaimeiniesta/metainspector)
MetaInspector/3.0.0 (+https://github.com/jaimeiniesta/metainspector)
MetaInspector/4.7.2 (+https://github.com/jaimeiniesta/metainspector)
MetaInspector/5.4.0 (+https://github.com/jaimeiniesta/metainspector)
MetaTagRobot/0.2 (http://www.seocentro.com/tools/search-engines/metatag-analyzer.html)
MetaTagRobot/2.0 (http://www.check-domains.com/website-analysis/website-analyzer.php)
MetaTagRobot/2.1 (http://www.seocentro.com/tools/search-engines/metatag-analyzer.html)
MetaURI API +metauri.com
MetaURI API/2.0 +metauri.com
Metaspinner/0.01 (Metaspinner; http://www.meta-spinner.de/; support@meta-spinner.de/)
Metaspinner/1.0 (Metaspinner Search Engine; http://www.meta-spinner.de/; support@meta-spinner.de)
MiaDev/0.0.1 (MIA Bot for research project MIA (www.MIA-marktplatz.de); http://www.mia-marktplatz.de/spider; spider@mia-marktplatz.de)
Microsearch.ru Bot 1.3;http://microsearch.ru/webmasters
Microsearch.ru/1.0; http://microsearch.ru/webmasters
Microsoft Bing Mobile SocialStreams Bot
Microsoft MSN SocialStreams Bot,gzip(gfe),gzip(gfe)
Microsoft.Data.Mashup (https://go.microsoft.com/fwlink/?LinkID=304225)
MicrosoftPrototypeCrawler (How's my crawling? mailto:newbiecrawler@hotmail.com)
Miniflux (http://miniflux.net)
Miniflux (https://miniflux.net)
Miro-HttpClient/1.0
MixBot (+http://t.co/GSRLLKex24)
Mixmax-LinkPreview 1.0 (+https://sdk.mixmax.com)
Mixnode/3.0.1 (+https://www.mixnode.com/; Speed analysis; no data is stored)
MixnodeCache/1.8(+https://cache.mixnode.com/)
MnoGoSearch/3.3.12
MnoGoSearch/3.3.15
MnoGoSearch/3.3.9
Mnogosearch-3.1.21
MojeekBot/0.2 (archi; http://www.mojeek.com/bot.html)
Mojoo Robot (http://www.mojoo.com/)
MonitorBacklinksWP (+http://monitorbacklinks.com/blog/incoming-links/)
MonkeyCrawl/0.05 (MonkeyCrawl; http://www.monkeymethods.org; )
MoodleBot/1.0
More-Testing (+http://www.theinternetrules.com)
Moreoverbot/5.1 ( http://w.moreover.com; webmaster@moreover.com) Mozilla/5.0
Morning Paper 1.0 (robots.txt compliant!)
Motoricerca-Robots.txt-Checker/1.0 (http://tool.motoricerca.info/robots-checker.phtml)
Mozila/5.0 (compatible; StudioFACA Search; +http://www.studiofaca.com/)
Mozilla (+http://web.nli.org.il/sites/NLI/English/collections/israel-collection/internet_archive/Pages/default.aspx)
Mozilla / reddit pic scraper v0.8 (bklug@tyx.net)
Mozilla 4.0 - GetMeLinked Spider www.GetMeLinked.com Web Directory
Mozilla 4.0(compatible; BotSeer/1.0; http://botseer.ist.psu.edu)
Mozilla crawl/5.0 (compatible; fairshare.cc +http://fairshare.cc)
Mozilla-Firefox-Spider(Wenanry)
Mozilla/2.0 (compatible; ARCHIVE.ORG.UA crawler)
Mozilla/2.0 (compatible; Ask Jeeves/Teoma)
Mozilla/2.0 (compatible; Ask Jeeves/Teoma; +http://about.ask.com/en/docs/about/webmasters.shtml)
Mozilla/2.0 (compatible; Ask Jeeves/Teoma; +http://sp.ask.com/docs/about/tech_crawling.html)
Mozilla/3.0 (compatible; Fluffy the spider; http://www.searchhippo.com/; info@searchhippo.com)
Mozilla/3.0 (compatible; ScollSpider; http://www.webwobot.com)
Mozilla/3.0 (compatible; WebMon 1.0.11; Windows XP)
Mozilla/4.0 (CMS Crawler: http://www.cmscrawler.com)
Mozilla/4.0 (NimbleCrawler 0.1) obeys NimbleCrawler directive contact jpump@looksmart.net
Mozilla/4.0 (SKIZZLE! Distributed Internet Spider v1.0 - www.SKIZZLE.com)
Mozilla/4.0 (Toread-Crawler/1.1; +http://news.toread.cc/crawler.php)
Mozilla/4.0 (compatible); MSIE 5.0; Medialab Spider
Mozilla/4.0 (compatible: AstraSpider V.2.1 : astrafind.com)
Mozilla/4.0 (compatible: FDSE robot)
Mozilla/4.0 (compatible: Shoula robot)
Mozilla/4.0 (compatible;  MSIE 5.01; GomezAgent 2.0; Windows NT)
Mozilla/4.0 (compatible;  Vagabondo/4.0/EU; http://webagent.wise-guys.nl/)
Mozilla/4.0 (compatible;  Vagabondo/4.0; http://webagent.wise-guys.nl/)
Mozilla/4.0 (compatible;  Vagabondo/4.0; http://webagent.wise-guys.nl/; http://www.wise-guys.nl/)
Mozilla/4.0 (compatible;  Vagabondo/4.0; webcrawler at wise-guys dot nl; http://webagent.wise-guys.nl/)
Mozilla/4.0 (compatible;  Vagabondo/4.0; webcrawler at wise-guys dot nl; http://webagent.wise-guys.nl/; http://www.wise-guys.nl/)
Mozilla/4.0 (compatible;  Vagabondo/4.0Beta; webcrawler at wise-guys dot nl; http://webagent.wise-guys.nl/; http://www.wise-guys.nl/)
Mozilla/4.0 (compatible; Advanced Email Extractor v2.61)
Mozilla/4.0 (compatible; BOTW Spider; +http://botw.org)
Mozilla/4.0 (compatible; BOTW Spider; http://botw.org)
Mozilla/4.0 (compatible; BlitzBot)
Mozilla/4.0 (compatible; DAWINCI ANTIPLAG SPIDER)
Mozilla/4.0 (compatible; FastCrawler3, support-fastcrawler3@fast.no)
Mozilla/4.0 (compatible; Fooooo_Web_Video_Crawl http://fooooo.com/bot.html)
Mozilla/4.0 (compatible; Globel; Traffic Sent From: http://addictive.me.uk)
Mozilla/4.0 (compatible; HostTracker.com/1.0;+http://host-tracker.com/)
Mozilla/4.0 (compatible; IDwhois/2.0alpha-10102619; Linux; en-US; +http://idwhois.info/about)
Mozilla/4.0 (compatible; Inspyder-Crawler; http://www.inspyder.com)
Mozilla/4.0 (compatible; Iplexx Spider/1.1)
Mozilla/4.0 (compatible; LinkMarket-Bot)
Mozilla/4.0 (compatible; MSIE 4.01; Windows CE; PPC; 240x320; SPV M700; OpVer 19.123.2.733) OrangeBot-Mobile 2008.0 (mobilesearch.support@orange-ftgroup.com)
Mozilla/4.0 (compatible; MSIE 4.0; MSIECrawler; Windows 95)
Mozilla/4.0 (compatible; MSIE 5.01; Windows NT 5.0; DomainDB-1.1; http://domaindb.com/crawler/)
Mozilla/4.0 (compatible; MSIE 5.01; Windows NT 5.0; FeedFinder-2.0; http://bloggz.se/crawler)
Mozilla/4.0 (compatible; MSIE 5.0; Windows 95) VoilaBot BETA 1.2 (http://www.voila.com/)
Mozilla/4.0 (compatible; MSIE 5.0; Windows 98; DigExt; Crayon Crawler)
Mozilla/4.0 (compatible; MSIE 5.0; Windows NT; Girafabot; girafabot at girafa dot com; http://www.girafa.com)
Mozilla/4.0 (compatible; MSIE 5.5; SuperSpider/139; Windows 98; Win 9x 4.90)
Mozilla/4.0 (compatible; MSIE 5.5; Windows 98; Infocrawler; Alexa Toolbar)
Mozilla/4.0 (compatible; MSIE 5.5; Windows NT 4.0; obot)
Mozilla/4.0 (compatible; MSIE 5.5; Windows NT 5.0) METASpider
Mozilla/4.0 (compatible; MSIE 5.5; Windows NT 5.0; mowser; http://www.mowser.com)
Mozilla/4.0 (compatible; MSIE 6.0 compatible; Asterias Crawler v4; http://www.singingfish.com/help/spider.html; webmaster@singingfish.com); SpiderThread Revision: 3.0
Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 4.0; Girafabot; girafabot at girafa dot com; http://www.girafa.com)
Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0) (larbin2.6.3@unspecified.mail)
Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0) (larbin@unspecified.mail)
Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0; WordSurfer Spider 2.2;))
Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0; baidu Transcoder;)
Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0; http://basicstate.com/)
Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0; http://www.pregnancycrawler.com)
Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0;.NET CLR 1.0.3705; ContextAd Bot 1.0)
Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; Netcraft SSL Server Survey - contact info@netcraft.com)
Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1 + FairShare-http://fairshare.cc)
Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1;  http://www.changedetection.com/bot.html )
Mozilla/4.0 (compatible; MSIE 6.0; Windows compatible LesnikBot)
Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.0) LinkCheck by Siteimprove.com
Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.0) Match by Siteimprove.com
Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.0) SiteCheck-sitecrawl by Siteimprove.com
Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; QQDownload 1.7; .NET CLR 2.0.50727)
Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; RetrevoPageAnalyzer; +http://www.retrevo.com/content/about-us)
Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; Trident/4.0; Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1) ; Embedded Web Browser from: http://bsalsa.com/; RSScrawler/4.0 (compatible; MSIE 6.0; Windows NT 5.0); .NET CLR 2.0.50727
Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.0) AddSugarSpiderBot www.idealobserver.com
Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.1; Trident/4.0; QQDownload 663; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; .NET4.0C; .NET CLR 1.1.4322)
Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.2; WOW64; Trident/6.0; .NET4.0E; .NET4.0C; .NET CLR 3.5.30729; .NET CLR 2.0.50727; .NET CLR 3.0.30729) CrawlerProcess (http://www.PowerMapper.com) /5.7.704.0
Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.2; WOW64; Trident/6.0; .NET4.0E; .NET4.0C; .NET CLR 3.5.30729; .NET CLR 2.0.50727; .NET CLR 3.0.30729) CrawlerProcess (http://www.PowerMapper.com) /5.9.726.0
Mozilla/4.0 (compatible; MSIE 7.0; acedo crawler extension)
Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0; .NET CLR 1.1.4322; .NET CLR 2.0.50727; .NET CLR 3.0.04506.30; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729; MDDS; PTST 2.386)
Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0; QQDownload 760; .NET CLR 2.0.50727; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729; InfoPath.2; .NET4.0E; .NET4.0C; .NET CLR 1.1.4322)
Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.2; Trident/4.0; .NET CLR 1.1.4322; .NET CLR 2.0.50727; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729; .NET4.0C; .NET4.0E; PTST 2.386)
Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.2; Trident/4.0; Data Center; .NET CLR 1.1.4322; .NET CLR 2.0.50727; .NET CLR 3.0.04506.648; .NET CLR 3.5.21022; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729; PTST 2.386)
Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.0; Trident/4.0; SLCC1; .NET CLR 2.0.50727; .NET CLR 1.1.4322; InfoPath.2; .NET CLR 3.5.21022; .NET CLR 3.5.30729; MS-RTC LM 8; OfficeLiveConnector.1.4; OfficeLivePatch.1.3; .NET CLR 3.0.30729) Jakarta Commons-HttpClient/3.0-rc3 PHPCrawl GStreamer souphttpsrc libsoup/2.27.4 PycURL/7.19.0 XML-RPC for PHP 2.2.1 GoogleFriendConnect/1.0 HTMLParser/1.6 gPodder/0.15.2 ( http://gpodder.org/) anw webtool LoadControl/1.3 WinHttp urlgrabber/3.1.0
Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.0; Trident/4.0; SLCC1; .NET CLR 2.0.50727; Media Center PC 5.0; .NET CLR 3.5.30729; .NET4.0C; .NET CLR 3.0.30729; .NET4.0E) PTST/276
Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.1; Trident/4.0; PTST 2.295)
Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.1; Trident/4.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; .NET4.0C; .NET4.0E; GWX:RED; PTST 2.386)
Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.1; Trident/4.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; .NET4.0C; .NET4.0E; PTST 2.386)
Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.1; Trident/4.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; MDDR; .NET4.0C; .NET4.0E; .NET CLR 1.1.4322; Tablet PC 2.0); 360Spider
Mozilla/4.0 (compatible; MuscatFerret/1.7; http://www.webtop.com/)
Mozilla/4.0 (compatible; NaverBot/1.0; http://help.naver.com/customer_webtxt_02.jsp)
Mozilla/4.0 (compatible; Netcraft Web Server Survey)
Mozilla/4.0 (compatible; SpeedySpider; www.entireweb.com)
Mozilla/4.0 (compatible; SpiderView 1.0;unix)
Mozilla/4.0 (compatible; Vagabondo/4.0/EU; http://www.wise-guys.nl/)
Mozilla/4.0 (compatible; Yet-Another-Spider; )
Mozilla/4.0 (compatible; bmdspider; windows 5.1)
Mozilla/4.0 (compatible; cron-job.org; http://cron-job.org/abuse/)
Mozilla/4.0 (compatible; focuseekbot)
Mozilla/4.0 (compatible; grub-client-0.3.0; Crawl your own stuff with http://grub.org)
Mozilla/4.0 (compatible; grub-client-1.0.4; Crawl your own stuff with http://grub.org)
Mozilla/4.0 (compatible; grub-client-1.0.5; Crawl your own stuff with http://grub.org)
Mozilla/4.0 (compatible; grub-client-1.0.6; Crawl your own stuff with http://grub.org)
Mozilla/4.0 (compatible; grub-client-1.0.7; Crawl your own stuff with http://grub.org)
Mozilla/4.0 (compatible; grub-client-1.1.1; Crawl your own stuff with http://grub.org)
Mozilla/4.0 (compatible; grub-client-1.2.1; Crawl your own stuff with http://grub.org)
Mozilla/4.0 (compatible; grub-client-1.3.1; Crawl your own stuff with http://grub.org)
Mozilla/4.0 (compatible; grub-client-1.3.7; Crawl your own stuff with http://grub.org)
Mozilla/4.0 (compatible; grub-client-1.4.3; Crawl your own stuff with http://grub.org)
Mozilla/4.0 (compatible; grub-client-1.5.3; Crawl your own stuff with http://grub.org)
Mozilla/4.0 (compatible; http://search.thunderstone.com/texis/websearch/about.html)
Mozilla/4.0 (compatible; lworld spider 1.0; Windows NT 5.1)
Mozilla/4.0 (compatible;HostTracker/2.0;+http://www.host-tracker.com/)
Mozilla/4.0 (vBSEO; http://www.vbseo.com)
Mozilla/4.0 compatible FurlBot/Furl Search 2.0 (FurlBot; http://www.furl.net; wn.furlbot@looksmart.net)
Mozilla/4.0 compatible ZyBorg/1.0 (wn-14.zyborg@looksmart.net; http://www.WISEnutbot.com)
Mozilla/4.0 compatible ZyBorg/1.0 (wn-16.zyborg@looksmart.net; http://www.WISEnutbot.com)
Mozilla/4.0 compatible ZyBorg/1.0 (wn.zyborg@looksmart.net; http://www.WISEnutbot.com)
Mozilla/4.0 compatible ZyBorg/1.0 Dead Link Checker (wn.dlc@looksmart.net; http://www.WISEnutbot.com)
Mozilla/4.0 compatible ZyBorg/1.0 Dead Link Checker (wn.zyborg@looksmart.net; http://www.WISEnutbot.com)
Mozilla/4.0, VM-Crawler/cs version info ofni noisrev sc
Mozilla/4.5 (compatible; HTTrack 3.0x; Windows 98)
Mozilla/5.0 ( Macintosh; Intel Mac OS X 10_10_1 ) AppleWebKit/600.2.5 ( KHTML, like Gecko ) Version/8.0.2 Safari/600.2.5 ( compatible; CloudServerMarketSpider/1.0; +http://cloudservermarket.com/spider.html )
Mozilla/5.0 ( compatible ; Veooz/1.0 ; +http://www.veooz.com/veoozbot.html )
Mozilla/5.0 ( compatible; SETOOZBOT/0.30 ; http://www.setooz.com/bot.html )
Mozilla/5.0 ( compatible; SETOOZBOT/0.30 ; http://www.setooz.com/bot.html ; agentname at setooz dot_com )
Mozilla/5.0 (+http://www.sli-systems.com/) Mammoth/0.1
Mozilla/5.0 (Android; Mobile; rv:21.0) Gecko/21.0 Firefox/21.0 commoncrawl.org/research//Nutch-1.7-SNAPSHOT
Mozilla/5.0 (Compatible; Vedma/0.91Beta; +http://www.vedma.ru/bot.htm)
Mozilla/5.0 (FauBot/0.1; +http://buzzvolume.com/fau/)
Mozilla/5.0 (GetLinkInfo.com - http://www.getlinkinfo.com)
Mozilla/5.0 (Google-PhysicalWeb)
Mozilla/5.0 (Grabify.link - https://grabify.link/expander)
Mozilla/5.0 (Java) outbrain
Mozilla/5.0 (KeepRight OpenStreetMap Checker; http://keepright.at) Gecko/20100101 Firefox/22.0
Mozilla/5.0 (Linux Android 5.0 SM-G900P Build/LRX21T) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/84.0.4147.125 Mobile Safari/537.36 Statically-Screenshot-Mobile/1.0 (+https://statically.io/screenshot/)
Mozilla/5.0 (Linux Android 5.0 SM-G900P Build/LRX21T) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/84.0.4147.125 Mobile Safari/537.36 Statically-Screenshot-PDF/1.0 (+https://statically.io/screenshot/)
Mozilla/5.0 (Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.79 Safari/537.36 (compatible; Statically-Screenshot; +https://statically.io/screenshot)
Mozilla/5.0 (Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/84.0.4147.125 Safari/537.36 Statically-Screenshot/1.0 (+https://statically.io/screenshot/)
Mozilla/5.0 (Linux; Android 4.0.4; Galaxy Nexus Build/IMM76B) AppleWebKit/537.36 (KHTML, like Gecko; Google-Publisher-Plugin) Chrome/27.0.1453 Mobile Safari/537.36
Mozilla/5.0 (Linux; Android 4.1.2; Galaxy Nexus Build/JZO54K; GTmetrix http://gtmetrix.com/) AppleWebKit/537.22 (KHTML, like Gecko) Chrome/26.0.1410.58 Mobile Safari/537.22
Mozilla/5.0 (Linux; Android 4.1.2; Galaxy Nexus Build/JZO54K; GTmetrix https://gtmetrix.com/) AppleWebKit/537.22 (KHTML, like Gecko) Chrome/26.0.1410.58 Mobile Safari/537.22
Mozilla/5.0 (Linux; Android 4.3; Galaxy Nexus Build/JWR66Y; GTmetrix https://gtmetrix.com/) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.68 Mobile Safari/537.36
Mozilla/5.0 (Linux; Android 4.4.3; HTC One Build/KTU84L) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/40.0.2125.111 Mobile Safari/537.36 DareBoost
Mozilla/5.0 (Linux; Android 4.4.3; HTC One Build/KTU84L) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840 Mobile Safari/537.36 DareBoost
Mozilla/5.0 (Linux; Android 4.4; Nexus 5 Build/KRT16M) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/40.0.2125.111 Mobile Safari/537.36 DareBoost
Mozilla/5.0 (Linux; Android 4.4; Nexus 5 Build/KRT16M) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/54.0.2840 Mobile Safari/537.36 DareBoost
Mozilla/5.0 (Linux; Android 5.0.2; SM-G920T Build/LRX22G) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/40.0.2125.111 Mobile Safari/537.36 DareBoost
Mozilla/5.0 (Linux; Android 5.0.2; SM-G920T Build/LRX22G) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840 Mobile Safari/537.36 DareBoost
Mozilla/5.0 (Linux; Android 5.0; LG-D855 Build/LRX21R.A1422018487) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/40.0.2214.109 Mobile Safari/537.36 DareBoost
Mozilla/5.0 (Linux; Android 5.0; LG-D855 Build/LRX21R.A1422018487) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840 Mobile Safari/537.36 DareBoost
Mozilla/5.0 (Linux; Android 5.0; Nexus 5 Build/LRX21O) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/46.0.2490.76 Mobile Safari/537.36 PTST/276
Mozilla/5.0 (Linux; Android 5.0; Nexus 5 Build/LRX21O) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/46.0.2490.76 Mobile Safari/537.36 PTST/281
Mozilla/5.0 (Linux; Android 5.0; SM-G900P Build/LRX21T) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/47.0.5267.1259 Mobile Safari/537.36; Bytespider
Mozilla/5.0 (Linux; Android 5.0; SM-G900P Build/LRX21T) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.7990.1979 Mobile Safari/537.36; Bytespider
Mozilla/5.0 (Linux; Android 5.0; SM-G900P Build/LRX21T) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.2268.1523 Mobile Safari/537.36; Bytespider
Mozilla/5.0 (Linux; Android 5.0; SM-G920A) AppleWebKit (KHTML, like Gecko) Chrome Mobile Safari (compatible; AdsBot-Google-Mobile; +http://www.google.com/mobile/adsbot.html)
Mozilla/5.0 (Linux; Android 6.0.1; Moto G (4) Build/MPJ24.139-64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.146 Mobile Safari/537.36 PTST/180829.190838
Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5 Build/MRA58N) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3694.0 Mobile Safari/537.36 Chrome-Lighthouse
Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5 Build/MRA58N) AppleWebKit/537.36(KHTML, like Gecko) Chrome/61.0.3116.0 Mobile Safari/537.36 Chrome-Lighthouse
Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5 Build/MRA58N) AppleWebKit/537.36(KHTML, like Gecko) Chrome/69.0.3464.0 Mobile Safari/537.36 Chrome-Lighthouse
Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/27.0.1453 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.96 Mobile Safari/537.36 (compatible; Google-AMPHTML)
Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.96 Mobile Safari/537.36 (compatible; Google-Read-Aloud; +https://support.google.com/webmasters/answer/1061943)
Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.96 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.96 Mobile Safari/537.36 (compatible; Pinterestbot/1.0;  https://www.pinterest.com/bot.html)
Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko; Google Page Speed Insights) Chrome/27.0.1453 Mobile Safari/537.36
Mozilla/5.0 (Linux; Android 6.0; Nexus 5 Build/MRA58N) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/40.0.2576.1836 Mobile Safari/537.36; Bytespider
Mozilla/5.0 (Linux; Android 6.0; Nexus 5 Build/MRA58N) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/43.0.9681.1227 Mobile Safari/537.36; Bytespider
Mozilla/5.0 (Linux; Android 6.0; Nexus 5 Build/MRA58N) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/44.0.6023.1635 Mobile Safari/537.36; Bytespider
Mozilla/5.0 (Linux; Android 6.0; Nexus 5 Build/MRA58N) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.0.4944.1981 Mobile Safari/537.36; Bytespider
Mozilla/5.0 (Linux; Android 6.0; Nexus 5 Build/MRA58N) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.3613.1739 Mobile Safari/537.36; Bytespider
Mozilla/5.0 (Linux; Android 6.0; Nexus 5 Build/MRA58N) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.4022.1033 Mobile Safari/537.36; Bytespider
Mozilla/5.0 (Linux; Android 6.0; Nexus 5 Build/MRA58N) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.5668.1446 Mobile Safari/537.36; Bytespider;bytespider@bytedance.com
Mozilla/5.0 (Linux; Android 6.0; Nexus 5 Build/MRA58N) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.0.3248.1547 Mobile Safari/537.36; Bytespider
Mozilla/5.0 (Linux; Android 6.0; Nexus 5 Build/MRA58N) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/57.0.5527.1507 Mobile Safari/537.36; Bytespider
Mozilla/5.0 (Linux; Android 7.0;) AppleWebKit/537.36 (KHTML, like Gecko) Mobile Safari/537.36 (compatible; AspiegelBot)
Mozilla/5.0 (Linux; Android 7.0;) AppleWebKit/537.36 (KHTML, like Gecko) Mobile Safari/537.36 (compatible; PetalBot;+https://aspiegel.com/petalbot)
Mozilla/5.0 (Linux; Android 7.0;) AppleWebKit/537.36 (KHTML, like Gecko) Mobile Safari/537.36 (compatible; PetalBot;+https://webmaster.petalsearch.com/site/petalbot)
Mozilla/5.0 (Linux; Android 8.0; Pixel 2 Build/OPD3.170816.012) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.5216.1326 Mobile Safari/537.36; Bytespider
Mozilla/5.0 (Linux; Android 8.0; Pixel 2 Build/OPD3.170816.012) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.9038.1080 Mobile Safari/537.36; Bytespider
Mozilla/5.0 (Linux; Android 8.0; Pixel 2 Build/OPD3.170816.012; DuplexWeb-Google/1.0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.131 Mobile Safari/537.36
Mozilla/5.0 (Linux; Android 8.0; Pixel 2 Build/OPD3.170816.012; Storebot-Google/1.0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/81.0.4044.138 Mobile Safari/537.36
Mozilla/5.0 (Linux; U; Android 1.6; ja-jp; SonyEricssonSO-01B Build/R1EA018) AppleWebKit/528.5 (KHTML, like Gecko) Version/3.1.2 Mobile Safari/525.20.1 (compatible; ichiro/mobile goo; http://help.goo.ne.jp/help/article/1142)
Mozilla/5.0 (Linux; U; Android 2.3.4; generic) AppleWebKit/537.36 (KHTML, like Gecko; Google Web Preview) Version/4.0 Mobile Safari/537.36
Mozilla/5.0 (Linux; U; Android 4.1.2; cs-cz; Seznam screenshot-generator Build/Q3) AppleWebKit/534.30 (KHTML, like Gecko) Version/4.0 Mobile Safari/534.30
Mozilla/5.0 (Linux;u;Android 2.3.7;zh-cn;) AppleWebKit/533.1 (KHTML,like Gecko) Version/4.0 Mobile Safari/533.1 (compatible; +http://www.baidu.com/search/spider.html)
Mozilla/5.0 (Macintosh; Butterfly/1.0; +http://labs.topsy.com/butterfly/) Gecko/2009032608 Firefox/3.0.8
Mozilla/5.0 (Macintosh; Intel Mac OS X 10.10; rv:41.0) Gecko/20100101 Firefox/55.0 BrandVerity/1.0 (http://www.brandverity.com/why-is-brandverity-visiting-me)
Mozilla/5.0 (Macintosh; Intel Mac OS X 10.11; rv:46.0) Gecko/20100101 Firefox/46.0 | Hardenize (https://www.hardenize.com)
Mozilla/5.0 (Macintosh; Intel Mac OS X 10.11; rv:46.0) Gecko/20100101 Firefox/46.0 | Hardenize/v1.1196.1 (https://www.hardenize.com)
Mozilla/5.0 (Macintosh; Intel Mac OS X 10.11; rv:48.0) Gecko/20100101 Firefox/48.0 glindahl-cocrawler/0.01 (+http://www.pbm.com/~lindahl/glindahl-cocrawler.html)
Mozilla/5.0 (Macintosh; Intel Mac OS X 10.11; rv:49.0) Gecko/20100101 Firefox/49.0 (FlipboardProxy/1.2; +http://flipboard.com/browserproxy)
Mozilla/5.0 (Macintosh; Intel Mac OS X 10.9; rv:28.0) Gecko/20100101 Firefox/28.0 (FlipboardProxy/1.1; +http://flipboard.com/browserproxy)
Mozilla/5.0 (Macintosh; Intel Mac OS X 10.9; rv:28.0) Gecko/20100101 Firefox/28.0 (FlipboardProxy/1.6; +http://flipboard.com/browserproxy)
Mozilla/5.0 (Macintosh; Intel Mac OS X 1068) AppleWebKit/534.50 (KHTML like Gecko) Version/5.1 Instapaper/4.0 (+http://www.instapaper.com/)
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10 _1) AppleWebKit/600.2.5 (KHTML, like Gecko) Version/8.0.2 Safari/600.2.5 (Applebot/0.1; +http://www.apple.com/go/applebot)
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/40.0.2214.111 Safari/537.36 moatbot
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/600.2.5 (KHTML, like Gecko) Version/8.0.2 Safari/600.2.5 (Amazonbot/0.1; +https://developer.amazon.com/support/amazonbot)
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/600.2.5 (KHTML, like Gecko) Version/8.0.2 Safari/600.2.5 (Applebot/0.1)
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/600.2.5 (KHTML, like Gecko) Version/8.0.2 Safari/600.2.5 (Applebot/0.1; +http://www.apple.com/go/applebot)
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/44.0.2403.125 Safari/537.36 (compatible; KosmioBot/1.0; +http://kosm.io/bot.html)
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/45.0.2454.101 Safari/537.36 SWIMGBot
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_6) AppleWebKit/537.36 (KHTML, like Gecko) Sparkler/0.2.0-SNAPSHOT
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.181 Safari/537.36 AppEngine-Google; (+http://code.google.com/appengine; appid: s~feedly-nikon3)
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.99 Safari/537.36 CloudflareDiagnostics/1.0
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) Project-Resonance (http://project-resonance.com/) (KHTML, like Gecko) Chrome/83.0.4103.61 Safari/537.36
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.88 Safari/537.36 (compatible; NameFo; https://namefo.com/)
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_6_8) AppleWebKit/537.13+ (KHTML, like Gecko) Version/5.1.7 Safari/534.57.2; Shoppimon Analyzer (http://www.shoppimon.com/)
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_2) AppleWebKit/535.7 (KHTML, like Gecko) Chrome/16.0.912.77 Safari/535.7 AppEngine-Google; (+http://code.google.com/appengine; appid: s~feedly-social)
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_2) AppleWebKit/535.7 (KHTML, like Gecko) Chrome/23.0.912.77 Safari/535.7 AppEngine-Google; (+http://code.google.com/appengine; appid: s~feedly-nikon3)
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_3) AppleWebKit/537.36 (KHTML, like Gecko, Google-Publisher-Plugin) Chrome/27.0.1453 Safari/537.36
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_2) AppleWebKit/537.13 (KHTML, like Gecko) Chrome/30.0.1599.66 Safari/537.13 Luminator-robots/2.0
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_5) AppleWebKit/536.30.1 (KHTML, like Gecko) (compatible; Startpagina-Linkchecker/1.0; +https://www.startpagina.nl/linkchecker)
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_2) AppleWebKit/600.1.4 (KHTML, like Gecko) Safari/600.1.4 (compatible; Princetonbot/1.0; +http://http://tigress-web.princeton.edu/~fy/bot.html)
Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.6; en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6 (FlipboardBrowserProxy/0.0.5; +http://flipboard.com/browserproxy)
Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.6; en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6 (FlipboardProxy/0.0.5; +http://flipboard.com/browserproxy)
Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.6; en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6 (FlipboardProxy/1.1; +http://flipboard.com/browserproxy)
Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.6; en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6 (FlipboardProxy/2.0; +http://flipboard.com/browserproxy)
Mozilla/5.0 (Mobile; rv:18.0) Gecko/18.0 Firefox/18.0 commoncrawl.org/research//Nutch-1.7-SNAPSHOT
Mozilla/5.0 (Nekstbot; http://nekst.ipipan.waw.pl/nekstbot/)
Mozilla/5.0 (Nekstbot; http://www.ipipan.waw.pl/nekst/nekstbot/)
Mozilla/5.0 (ShopAlike; LadenZeile) FeedBot
Mozilla/5.0 (TweetmemeBot/4.0; +http://datasift.com/bot.html) Gecko/20100101 Firefox/31.0
Mozilla/5.0 (Twiceler-0.9 http://www.cuil.com/twiceler/robot.html)
Mozilla/5.0 (URLfilterDB-crawler/1.1) ufdb/1.0
Mozilla/5.0 (Unknown; Linux x86_64) AppleWebKit/534.34 (KHTML, like Gecko) PhantomJS/1.9.7 Daum Raam Edition b20140820 Safari/534.34
Mozilla/5.0 (Unknown; Linux x86_64) AppleWebKit/534.34 (KHTML, like Gecko) PingdomTMS/0.8.5 Safari/534.34
Mozilla/5.0 (Unknown; Linux x86_64) AppleWebKit/538.1 (KHTML, like Gecko) PhantomJS/2.1.1 Safari/538.1 Prerender (+https://github.com/prerender/prerender)
Mozilla/5.0 (Unknown; Linux x86_64) AppleWebKit/538.1 (KHTML, like Gecko) PhantomJS/2.1.1 Safari/538.1 bl.uk_lddc_renderbot/2.0.0 (+ http://www.bl.uk/aboutus/legaldeposit/websites/websites/faqswebmaster/index.html)
Mozilla/5.0 (Votay bot/4.0; http://www.votay.com/arts/comics/)
Mozilla/5.0 (WhatsMyIP.org HTTP_Compression_Test) http://whatsmyip.org/ua
Mozilla/5.0 (WhatsMyIP.org HTTP_Headers) http://whatsmyip.org/ua
Mozilla/5.0 (WhatsMyIP.org Text_to_Code_Ratio_Tool) http://whatsmyip.org/ua
Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.84 Safari/537.36 DMBrowser-UV
Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/55.0.2883.87 Safari/537.36 (compatible; Sitebulb/1.1; +https://sitebulb.com)
Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/55.0.2883.87 Safari/537.36 PTST/348
Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/77.0.3865.56 Safari/537.36 DMBrowser/2.1 (UV)
Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3112.113 Safari/537.36 (+https://whatis.contentkingapp.com)
Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.75 Safari/537.36 (compatible; SMTBot/1.0; +http://www.similartech.com/smtbot)
Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.75 Safari/537.36 (compatible; SMTBot/1.0; http://www.similartech.com/smtbot)
Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.77 Safari/537.36 TryJsoup/1.0 (+http://try.jsoup.org/)
Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.97 Safari/537.36 The National Archives UK Government Web Archive: http://www.nationalarchives.gov.uk/webarchive/; webarchive@nationalarchives.gov.uk
Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) browser/2020.2.0 Chrome/78.0.3904.130 Electron/7.1.7 Safari/537.36 PingdomTMS/2020.2
Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) browser/2020.2.1 Chrome/78.0.3904.130 Electron/7.3.2 Safari/537.36 PingdomTMS/2020.2
Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) browser/2020.2.5 Chrome/78.0.3904.130 Electron/7.3.15 Safari/537.36 PingdomTMS/2020.2
Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) renderer/2020.2.0 Chrome/78.0.3904.130 Electron/7.1.7 Safari/537.36 PingdomTMS/2020.2
Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko; compatible; BW/1.1; bit.ly/2W6Px8S) Chrome/84.0.4147.105 Safari/537.36
Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko; compatible; Cookiebot/1.0;  http://cookiebot.com/) Chrome/79.0.3945.117 Safari/537.36
Mozilla/5.0 (Windows NT 10.0; Win64; x64; BDCbot/1.0; +http://bigweb.bigdatacorp.com.br/faq.aspx) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/69.0.3497.100 Safari/537.36
Mozilla/5.0 (Windows NT 10.0; Win64; x64; trendictionbot0.5.0; trendiction search; http://www.trendiction.de/bot; please let us know of any problems; web at trendiction.com) Gecko/20170101 Firefox/67.0
Mozilla/5.0 (Windows NT 5.1) AppleWebKit/535.1 (KHTML, like Gecko) Chrome/14.0.835.202 Safari/535.1 google_partner_monitoring FWSzVTDDBz14547302713138T
Mozilla/5.0 (Windows NT 5.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/49.0.2623.112 Safari/537.36 PTST/201
Mozilla/5.0 (Windows NT 5.1; U; Win64; fr; rv:1.8.1) VoilaBot BETA 1.2 (support.voilabot@orange-ftgroup.com)
Mozilla/5.0 (Windows NT 5.1; rv:11.0) Gecko Firefox/11.0 (via ggpht.com GoogleImageProxy)
Mozilla/5.0 (Windows NT 5.2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/49.0.2623.112 Safari/537.36 PTST/201
Mozilla/5.0 (Windows NT 6.0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/49.0.2623.112 Safari/537.36 PTST/276
Mozilla/5.0 (Windows NT 6.0; rv:45.0) Gecko/20100101 Firefox/45.0 PTST/276
Mozilla/5.0 (Windows NT 6.1) (compatible; SMTBot/1.0; +http://www.similartech.com/smtbot)
Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.1 (KHTML, like Gecko) Chrome/21.0.1180.89 Safari/537.1; 360Spider
Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.1 (KHTML, like Gecko) Chrome/21.0.1180.89 Safari/537.1; 360Spider(compatible; HaosouSpider; http://www.haosou.com/help/help_3_2.html)
Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.1 (KHTML, like Gecko) Chrome/21.0.1180.89 Safari/537.1; http://spyonweb.com spyonweb@gmail.com
Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.0.2661.102 Safari/537.36 PTST/277
Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.103 Safari/537.36 PTST/284
Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.79 Safari/537.36 (http://www.shrinktheweb.com)
Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.79 Safari/537.36 (https://shrinktheweb.com)
Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/53.0.2785.116 Safari/537.36 PTST/314
Mozilla/5.0 (Windows NT 6.1)+(compatible; SMTBot/1.0; +http://www.similartech.com/smtbot)
Mozilla/5.0 (Windows NT 6.1; Trident/7.0; rv:11.0; PTST 2.386) like Gecko
Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/534 (KHTML, like Gecko) BingPreview/1.0b
Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/534+ (KHTML, like Gecko) BingPreview/1.0b
Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/534+ (KHTML, like Gecko) MsnBot-Media /1.0b
Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/534.34 (KHTML, like Gecko) Safari/534.34; +http://sniptracker.com
Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/534.57.2 (KHTML, like Gecko) Version/5.1.7 Safari/534.57.2 PTST/276
Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/534.57.2 (KHTML, like Gecko) Version/5.1.7 Safari/534.57.2 PTST/277
Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/534.57.2 (KHTML, like Gecko) Version/5.1.7 Safari/534.57.2 PTST/278
Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/33.0.1750.146 Safari/537.36 Specificfeeds- http://www.specificfeeds.com
Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/40.0.2214.93 Safari/537.36 XaxisSemanticsClassifier/1.0 http://crystalsemantics.com
Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/44.0.2403.155 Safari/537.36 TinEye/1.0 (via http://www.tineye.com/)
Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/44.0.2403.157 Safari/537.36 TinEye/1.0 (via http://www.tineye.com/)
Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/45.0.2454.101 Safari/537.36 TinEye/1.0 (via http://www.tineye.com/)
Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/49.0.2623.112 Safari/537.36 PTST/276
Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.0.2661.102 Safari/537.36 PTST/277
Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.0.2661.102 Safari/537.36; 360Spider
Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.0.2661.75 Safari/537.36 PTST/276
Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.0.2661.87 Safari/537.36 PTST/276
Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.0.2661.94 Safari/537.36 PTST/276
Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.103 Safari/537.36 PTST/284
Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/53.0.2785.116 Safari/537.36 PTST/314
Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.99 Safari/537.36 PTST/345
Mozilla/5.0 (Windows NT 6.1; WOW64) SkypeUriPreview Preview/0.5
Mozilla/5.0 (Windows NT 6.1; WOW64) adbeat.com/policy AppleWebKit/537.36 (KHTML, like Gecko) Chrome/48.0.2564.116 Safari/537.36
Mozilla/5.0 (Windows NT 6.1; WOW64; Trident/7.0; rv:11.0) like Gecko PTST/276
Mozilla/5.0 (Windows NT 6.1; WOW64; Trident/7.0; rv:11.0) like Gecko PTST/277
Mozilla/5.0 (Windows NT 6.1; WOW64; Trident/7.0; rv:11.0; topster.de Linkchecker 5.0) like Gecko (194.228.205.74)
Mozilla/5.0 (Windows NT 6.1; WOW64; Trident/7.0; rv:11.0; topster.de Linkchecker 5.0) like Gecko (90.177.52.133)
Mozilla/5.0 (Windows NT 6.1; WOW64; rv:13.0; GomezAgent 3.0) Gecko/20100101 Firefox/13.0.1
Mozilla/5.0 (Windows NT 6.1; WOW64; rv:18.0) Gecko/20100101 Firefox/18.0 AppEngine-Google; (+http://code.google.com/appengine; appid: s~aeshortener)
Mozilla/5.0 (Windows NT 6.1; WOW64; rv:45.0) Gecko/20100101 Firefox/45.0 PTST/276
Mozilla/5.0 (Windows NT 6.1; WOW64; rv:46.0) Gecko/20100101 Firefox/46.0 PTST/277
Mozilla/5.0 (Windows NT 6.1; WOW64; rv:46.0) Gecko/20100101 Firefox/46.0 PTST/278
Mozilla/5.0 (Windows NT 6.1; WOW64; rv:46.0) Gecko/20100101 Firefox/46.0 PTST/279
Mozilla/5.0 (Windows NT 6.1; WOW64; rv:48.0) Gecko/20100101 Firefox/48.0 PTST/314
Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.21 (KHTML, like Gecko) adspider Safari/537.21
Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/55.0.2883.87 Safari/537.36 TinEye/1.0 (via http://www.tineye.com/)
Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/69.0.3497.81 YisouSpider/5.0 Safari/537.36
Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/84.0.4147.89 Safari/537.36 MicroAdBot/1.1 (https://www.microad.co.jp/contact/)
Mozilla/5.0 (Windows NT 6.1; Win64; x64) KomodiaBot/1.0
Mozilla/5.0 (Windows NT 6.1; Win64; x64; +http://www.komodia.com/newwiki/index.php/URL_server_crawler) KomodiaBot/1.0
Mozilla/5.0 (Windows NT 6.1; Win64; x64; +https://www.ubtsupport.com/legal/Streamline3Bot.php) Streamline3Bot/1.0
Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:25.0; https://app.limber.io) Gecko/20100101 Firefox/25.0
Mozilla/5.0 (Windows NT 6.1; compatible; BDCbot/1.0; +http://bigweb.bigdatacorp.com.br/faq.aspx) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.118 Safari/537.36
Mozilla/5.0 (Windows NT 6.1; compatible; BDCbot/1.0; +http://ecommerce.bigdatacorp.com.br/faq.aspx) AppleWebKit/535.1 (KHTML, like Gecko) Chrome/13.0.782.107 Safari/535.1
Mozilla/5.0 (Windows NT 6.1; compatible; BDCbot/1.0; +http://ecommerce.bigdatacorp.com.br/faq.aspx) ppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.118 Safari/537.36
Mozilla/5.0 (Windows NT 6.1; rv:34.0) Gecko/20100101 Firefox/34.0; Dragonbot; http://www.dragonmetrics.com
Mozilla/5.0 (Windows NT 6.1; rv:38.0) Gecko/20100101 Firefox/38.0 (IndeedBot 1.1)
Mozilla/5.0 (Windows NT 6.1; rv:6.0) Gecko/20110814 Firefox/6.0 Google (+https://developers.google.com/+/web/snippet/)
Mozilla/5.0 (Windows NT 6.1; rv:6.0) Gecko/20110814 Firefox/6.0 Google favicon
Mozilla/5.0 (Windows NT 6.2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/31.0.1650.63 Safari/537.36 QIHU 360SE; 360Spider
Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/537.15 (KHTML like Gecko; +http://www.diffbot.com) Chrome/24.0.1295.0 Safari/537.15
Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/537.22 (KHTML, like Gecko) Chrome/25.0.1364.5 Safari/537.22 +awesomecrawler
Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/537.4 (KHTML, like Gecko) Chrome/98 Safari/537.4 (StatusCake SSL Monitor)
Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/537.4 (KHTML, like Gecko) Chrome/98 Safari/537.4 (StatusCake)
Mozilla/5.0 (Windows NT 6.2; WOW64) Runet-Research-Crawler (itrack.ru/research/cmsrate; rating@itrack.ru)
Mozilla/5.0 (Windows NT 6.2; WOW64) Russian CMS rating crawler (itrack.ru/cmsrate, avlasov@itrack.ru)
Mozilla/5.0 (Windows NT 6.2; WOW64; rv:29.0) Gecko/20100101 /29.0 DMBrowser-BV
Mozilla/5.0 (Windows NT 6.3; Trident/7.0; rv 11.0) like Gecko (compatible; Zombiebot/2.1; +http://www.zombiedomain.net/robot/)
Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.2311.90 Safari/537.36 PTST/276
Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.2311.90 Safari/537.36 PTST/277
Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/49.0.2623.112 Safari/537.36 PTST/276
Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.0.2661.102 Safari/537.36 PTST/277
Mozilla/5.0 (Windows NT 6.3; WOW64; Trident/7.0; rv:11.0) like Gecko PTST/276
Mozilla/5.0 (Windows NT 6.3; WOW64; Trident/7.0; rv:11.0; BingPreview/1.0b) like Gecko
Mozilla/5.0 (Windows NT 6.3; WOW64; Trident/7.0; rv:11.0; Catchpoint) like Gecko
Mozilla/5.0 (Windows NT 6.3; WOW64; rv:45.0) Gecko/20100101 Firefox/45.0 PTST/276
Mozilla/5.0 (Windows NT 6.3; WOW64; rv:46.0) Gecko/20100101 Firefox/46.0 PTST/277
Mozilla/5.0 (Windows NT 6.3; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/77.0.3865.120 Safari/537.36 (compatible; PagePeeker/3.0; +https://pagepeeker.com/robots/)
Mozilla/5.0 (Windows NT 6.3;compatible; Leikibot/1.0; +http://www.leiki.com)
Mozilla/5.0 (Windows Phone 8.1; ARM; Trident/7.0; Touch; rv:11.0; IEMobile/11.0; NOKIA; Lumia 530) like Gecko (compatible; adidxbot/2.0; +http://www.bing.com/bingbot.htm)
Mozilla/5.0 (Windows; U; MSIE 9.0; Windows NT 9.0; en-US) AppEngine-Google; (+http://code.google.com/appengine; appid: s~virustotalcloud)
Mozilla/5.0 (Windows; U; Win98; en-US; rv:1.8webcrawler) http://skateboarddirectory.com
Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:1.7.10) Gecko/20050716 Thunderbird/1.0.6 - WebCrawler http://cognitiveseo.com/bot.html
Mozilla/5.0 (Windows; U; Windows NT 5.1; de; rv:1.9.0.7; Google-SearchByImage) Gecko/2009021910 Firefox/3.0.7
Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/27.0.1453.116 Safari/537.36 HubSpot Marketing Grader
Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/27.0.1453.116 Safari/537.36 HubSpot Webcrawler
Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/27.0.1453.116 Safari/537.36 HubSpot Webcrawler - web-crawlers@hubspot.com
Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US) Speedy Spider (http://www.entireweb.com/about/search_tech/speedy_spider/)
Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US) Speedy Spider for SpeedyAds (http://www.entireweb.com/about/search_tech/speedy_spider/)
Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.7) Gecko/20040707 Lightningspider/0.9.2
Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.6) Gecko/20070725 Firefox/2.0.0.6 - James BOT - WebCrawler http://cognitiveseo.com/bot.html
Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.6) Gecko/20070725 Firefox/2.0.0.6 Ara.com.tr AraBot 1.0
Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9) Gecko/2008052906 Firefox/3.0 (BrowserSpyBot/1.0) (http://browserspy.dk/)
Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1.2) Gecko/20090729 Firefox/3.5.2 (.NET CLR 3.5.30729; Diffbot/0.1; +http://www.diffbot.com)
Mozilla/5.0 (Windows; U; Windows NT 5.1; en; rv:1.9.0.13) Gecko/2009073022 Firefox/3.5.2 (.NET CLR 3.5.30729) SurveyBot/2.3 (DomainTools)
Mozilla/5.0 (Windows; U; Windows NT 5.1; fr; rv:1.8.1) VoilaBot BETA 1.2 (support.voilabot@orange-ftgroup.com)
Mozilla/5.0 (Windows; U; Windows NT 5.1; pl; PageThing http://pagething.com); rv:1.9; Gecko/2008052906 Firefox/3.0
Mozilla/5.0 (Windows; U; Windows NT 5.1; zh-CN; )  Firefox/1.5.0.11; 360Spider
Mozilla/5.0 (Windows; U; Windows NT 5.1; zh-CN; rv:1.8.0.11)  Firefox/1.5.0.11; 360Spider
Mozilla/5.0 (Windows; U; Windows NT 5.1; zh-CN; rv:1.8.0.11) Firefox/1.5.0.11 360Spider;
Mozilla/5.0 (Windows; U; Windows NT 5.1; zh-CN; rv:1.8.0.11) Gecko/20070312 Firefox/1.5.0.11; 360Spider
Mozilla/5.0 (Windows; U; Windows NT 5.1; zh-CN; rv:1.9.2.8) Firefox/3.6.8/Nutch-1.7
Mozilla/5.0 (Windows; U; Windows NT 5.1; zh-CN; rv:1.9.2.8;baidu Transcoder) Gecko/20100722 Firefox/3.6.8 ( .NET CLR 3.5.30729)
Mozilla/5.0 (Windows; U; Windows NT 5.1;fr;rv:1.8.1) VoilaBotCollector BETA 0.1  (http://www.voila.com/)
Mozilla/5.0 (Windows; U; Windows NT 5.1;fr;rv:1.8.1) VoilaBotCollector BETA 0.1 (http://www.voila.com/)
Mozilla/5.0 (Windows; U; Windows NT 6.0; en-GB; rv:1.0; trendictionbot0.4.2; trendiction media ssppiiddeerr; http://www.trendiction.com/bot/; please let us know of any problems; ssppiiddeerr at trendiction.com) Gecko/20071127 Firefox/2.0.0.11
Mozilla/5.0 (Windows; U; Windows NT 6.0; en-GB; rv:1.0; trendictionbot0.4.5; trendiction search; http://www.trendiction.de/bot; please let us know of any problems; web at trendiction.com) Gecko/20071127 Firefox/3.0.0.11
Mozilla/5.0 (Windows; U; Windows NT 6.0; en-GB; rv:1.0; trendictionbot0.5.0; trendiction search; http://www.trendiction.de/bot; please let us know of any problems; web at trendiction.com) Gecko/20071127 Firefox/3.0.0.11
Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US; aggregator VocusBot 0.4; +http://www.vocus.com/vnhs.html)
Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US; rv:1.9.0.4) Gecko/2008102920 http://ow.ly web crawler (.NET CLR 3.5.30729)
Mozilla/5.0 (Windows; U; Windows NT 6.1; en; rv:1.9.1.3; MySpaceScraper) Gecko/20090824 Firefox/3.5.3 (.NET CLR 3.5.30729)
Mozilla/5.0 (Windows; Windows NT 6.1; WOW64; rv:2.0b8pre; .NET CLR 1.1.4322; .NET CLR 2.0.50727; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729; MS-RTC LM 8; OfficeLiveConnector.1.4; OfficeLivePatch.1.3; SLCC1; SLCC2; Media Center PC 6.0; GTB6.4; InfoPath.2; en-US; FunWebProducts; Zango 10.1.181.0; SV1; PRTG Network Monitor (www.paessler.com)) Gecko/20101114 Firefox/4.0b8pre QuickTime/7.6.2 Songbird/1.1.2 Web-sniffer/1.0.36 lftp/3.7.4 libwww-perl/5.820 GSiteCrawler/v1.12 rev. 260 Snoopy v1.2
Mozilla/5.0 (Windows;) NimbleCrawler 2.0.1 obeys UserAgent NimbleCrawler For problems contact: crawler@healthline.com
Mozilla/5.0 (X11; Datanyze; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.181 Safari/537.36
Mozilla/5.0 (X11; Linux i686 on x86_64; rv:25.0; GTmetrix http://gtmetrix.com/) Gecko/20100101 Firefox/25.0
Mozilla/5.0 (X11; Linux i686) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/30.0.1599.101 Safari/537.36; SSL-Crawler: http://crawler.dcsec.uni-hannover.de
Mozilla/5.0 (X11; Linux x86_64 UXCrawlerBot http://zium.co)
Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/534.24 (KHTML, like Gecko; Google Web Preview) Chrome/11.0.696 Safari/534.24
Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/534.34 (KHTML, like Gecko) Chromium/31.0.1650.63 Site-Shot/2.1 (http://www.site-shot.com/) Safari/534.34
Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/534.34 (KHTML, like Gecko) Qt/4.8.3 Safari/534.34 https://linkpeek.com
Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/534.34 (KHTML, like Gecko) Site-Shot/2.1 (http://www.site-shot.com/) Safari/534.34
Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/534.51 (KHTML, like Gecko; Google Web Preview) Chrome/12.0.742 Safari/534.51
Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/535. Safari/535.22+; (+http://immediatenet.com/thumbnail_api.html; free thumbnails)
Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/535.21 (KHTML, like Gecko) Chrome/19.0.1042.0 Safari/535.21 PagePeeker/2.1; +http://pagepeeker.com/robots/
Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/27.0.1453 Safari/537.36 (compatible; Google-HotelAdsVerifier/2.0)
Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/40.0.2125.111 Safari/537.36 DareBoost
Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2228.0 Safari/537.36 (compatible; YandexScreenshotBot/3.0; +http://yandex.com/bots)
Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.118 Safari/537.36 (compatible; Google-Read-Aloud; +https://support.google.com/webmasters/answer/1061943)
Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/43.0.2357.134 Safari/537.36 http://notifyninja.com/monitoring
Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/49.0.2623.75 Safari/537.36 Google (+https://developers.google.com/+/web/snippet/)
Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/49.0.2623.75 Safari/537.36 Google Favicon
Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.63 Safari/537.36 (compatible; aylienbot/0.2; +http://www.aylien.com/bot.html)
Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840 Safari/537.36 DareBoost
Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/62.0.3202.75 Safari/537.36 DareBoost
Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3694.0 Safari/537.36 Chrome-Lighthouse
Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.45 Safari/537.36 PTST/211202.211915
Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/66.0.3336.0 Safari/537.36
Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/69.0.3494.0 Safari/537.36
Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/71.0.3578.98 Safari/537.36 Prerender (+https://github.com/prerender/prerender)
Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/74.0.3729.169 Safari/537.36
Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/76.0.3803.0 Safari/537.36
Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Ubuntu Chromium/59.0.3071.109 Chrome/59.0.3071.109 Safari/537.36 PingdomPageSpeed/1.0 (pingbot/2.0; +http://www.pingdom.com/)
Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Ubuntu Chromium/61.0.3163.100 Chrome/61.0.3163.100 Safari/537.36 PingdomPageSpeed/1.0 (pingbot/2.0; +http://www.pingdom.com/)
Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Ubuntu Chromium/61.0.3163.100 Chrome/61.0.3163.100 Safari/537.36 PingdomPageSpeed/1.0 (pingbot/2.0; http://www.pingdom.com/)
Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko; Google Page Speed Insights) Chrome/27.0.1453 Safari/537.36
Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko; Google Search Console) Chrome/27.0.1453 Safari/537.36
Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko; Google Search Console) Chrome/41.0.2272.118 Safari/537.36
Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko; Google Web Preview) Chrome/27.0.1453 Safari/537.36
Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36(KHTML, like Gecko) Chrome/69.0.3464.0 Safari/537.36 Chrome-Lighthouse
Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.4 (KHTML, like Gecko; Google Page Speed Insights) Chrome/22.0.1229 Safari/537.4
Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.4 (KHTML, like Gecko; Google Web Preview) Chrome/22.0.1229 Safari/537.4
Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/538.1 (KHTML, like Gecko) Chromium/31.0.1650.63 Site-Shot/2.1 (http://www.site-shot.com/) Safari/538.1
Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/538.1 (KHTML, like Gecko) JobBot/5.0 (compatible; +http://www.jobseeker.com.au/bot.html) Safari/538.1
Mozilla/5.0 (X11; Linux x86_64) adbeat.com/policy AppleWebKit/537.21 (KHTML, like Gecko) Firefox/34.0 Safari/537.21
Mozilla/5.0 (X11; Linux x86_64)AppleWebKit/537.36 (KHTML, like Gecko; Google-Ads-Qualify)Chrome/85.0.4183.140 Safari/537.36
Mozilla/5.0 (X11; Linux x86_64; GTmetrix http://gtmetrix.com/) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/31.0.1650.63 Safari/537.36
Mozilla/5.0 (X11; Linux x86_64; GTmetrix https://gtmetrix.com/) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.0.2661.75 Safari/537.36
Mozilla/5.0 (X11; Linux x86_64; GTmetrix https://gtmetrix.com/) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.106 Safari/537.36
Mozilla/5.0 (X11; Linux x86_64; GTmetrix https://gtmetrix.com/) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.71 Safari/537.36
Mozilla/5.0 (X11; Linux x86_64; GTmetrix https://gtmetrix.com/) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/75.0.3770.100 Safari/537.36
Mozilla/5.0 (X11; Linux x86_64; HubSpot Single Page link check; web-crawlers+links@hubspot.com) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.103 Safari/537.36
Mozilla/5.0 (X11; Linux x86_64; rv:35.0) Gecko/20100101 Firefox/35.0 DareBoost
Mozilla/5.0 (X11; Linux x86_64; rv:39.0; GTmetrix http://gtmetrix.com/) Gecko/20100101 Firefox/39.0
Mozilla/5.0 (X11; Linux x86_64; rv:41.0; GTmetrix https://gtmetrix.com/) Gecko/20100101 Firefox/41.0
Mozilla/5.0 (X11; Linux x86_64; rv:43.0; GTmetrix https://gtmetrix.com/) Gecko/20100101 Firefox/43.0
Mozilla/5.0 (X11; Linux x86_64; rv:45.0; GTmetrix https://gtmetrix.com/) Gecko/20100101 Firefox/45.0
Mozilla/5.0 (X11; Linux x86_64; rv:47.0; GTmetrix https://gtmetrix.com/) Gecko/20100101 Firefox/47.0
Mozilla/5.0 (X11; Linux x86_64; rv:49.0; GTmetrix https://gtmetrix.com/) Gecko/20100101 Firefox/49.0
Mozilla/5.0 (X11; U; Linux Core i7-4980HQ; de; rv:32.0; compatible; Jobboerse.com; http://www.xn--jobbrse-d1a.com) Gecko/20100401 Firefox/24.0
Mozilla/5.0 (X11; U; Linux Core i7-4980HQ; de; rv:32.0; compatible; JobboerseBot; http://www.jobboerse.com/bot.htm) Gecko/20100101 Firefox/38.0
Mozilla/5.0 (X11; U; Linux Core i7-4980HQ; de; rv:32.0; compatible; JobboerseBot; https://www.jobboerse.com/bot.htm) Gecko/20100101 Firefox/38.0
Mozilla/5.0 (X11; U; Linux Core i7-4980HQ; de; rv:32.0; compatible; JobboerseBot; https://www.jobboerse.com/bot.htm) Gecko/20100401 Firefox/24.0
Mozilla/5.0 (X11; U; Linux i686 (x86_64); en-US; rv:1.8.1.11) Gecko/20080109 (Charlotte/0.9t; http://www.searchme.com/support/) (Charlotte/0.9t; http://www.searchme.com/support/)
Mozilla/5.0 (X11; U; Linux i686 (x86_64); en-US; rv:1.8.1.20) Gecko/20090429 HeartRails_Capture/0.6 (+http://capture.heartrails.com/) BonEcho/2.0.0.20
Mozilla/5.0 (X11; U; Linux i686; de; rv:1.9.0.1; compatible; iCjobs Stellenangebote Jobs; http://www.icjobs.de) Gecko/20100401 iCjobs/3.2.3
Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.2.1; aggregator:Spinn3r (Spinn3r 3.1); http://spinn3r.com/robot) Gecko/20021130
Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.7.12; ips-agent) Gecko/20050922 Fedora/1.0.7-1.1.fc4 Firefox/1.0.7
Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.0.16) Gecko/2009121601 Ubuntu/9.04 (jaunty) Firefox/3.0.16 Specificfeeds- http://www.specificfeeds.com
Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.1.3; ips-agent) Gecko/20090824 Fedora/1.0.7-1.1.fc4  Firefox/3.5.3
Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.2.17) Gecko/20110515 HeartRails_Capture/1.0.4 (+http://capture.heartrails.com/) Namoroka/3.6.17
Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.2.18) Gecko/20110622 HeartRails_Capture/1.0.4 (+http://capture.heartrails.com/) Namoroka/3.6.18
Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.2.18) Gecko/20110622 HeartRails_Capture/1.0.5 (+http://capture.heartrails.com/) Namoroka/3.6.18
Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.2.18) Gecko/20130331 HeartRails_Capture/1.0.5 (+http://capture.heartrails.com/) Namoroka/3.6.18
Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.2.24; ips-agent) Gecko/20111107 Ubuntu/10.04 (lucid) Firefox/3.6.24
Mozilla/5.0 (X11; U; Linux i686; xx; rv:1.9.2.18) Gecko/20110622 HeartRails_Capture/1.0.5 (+http://capture.heartrails.com/) Namoroka/3.6.18
Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.0.19; aggregator:Spinn3r (Spinn3r 3.1); http://spinn3r.com/robot) Gecko/2010040121 Firefox/3.0.19
Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.0.5) Gecko/2010033101 Gentoo Firefox/3.0.5 (Dot TK - spider 3.0)
Mozilla/5.0 (X11; Ubuntu; Linux i686; rv:14.0; ips-agent) Gecko/20100101 Firefox/14.0.1
Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:14.0) Gecko/20100101 Firefox/14.0.1 AppEngine-Google; (+http://code.google.com/appengine; appid: s~metacdn-hr)
Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:49.0) Gecko/20100101 Firefox/49.0 / GnowitNewsbot / Contact information at http://www.gnowit.com
Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:66.0) WhatCMS/1.0
Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:94.0) Gecko/20100101 Firefox/94.0 PTST/211202.211915
Mozilla/5.0 (X11; compatible; semantic-visions.com crawler; HTTPClient 3.1)
Mozilla/5.0 (Yahoo-MMCrawler/4.0; mailto:vertical-crawl-support@yahoo-inc.com)
Mozilla/5.0 (YahooYSMcm/3.0.0; http://help.yahoo.com)
Mozilla/5.0 (bl.uk_lddc_bot; Linux x86_64) PhantomJS/1.9.7 (+http://www.bl.uk/aboutus/legaldeposit/websites/websites/faqswebmaster/index.html)
Mozilla/5.0 (compatibl$; Miniflux/2.0.x-dev; +https://miniflux.app)
Mozilla/5.0 (compatible) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/47.0.2526.73 Safari/537.36 collection@infegy.com
Mozilla/5.0 (compatible) Feedfetcher-Google; (+http://www.google.com/feedfetcher.html)
Mozilla/5.0 (compatible) Feedfetcher-Google;(+http://www.google.com/feedfetcher.html)
Mozilla/5.0 (compatible) SemanticScholarBot (+https://www.semanticscholar.org/crawler)
Mozilla/5.0 (compatible) WatchSumo/1.0.0 (http://www.watchsumo.com)
Mozilla/5.0 (compatible;  Page2RSS/0.7; +http://page2rss.com/)
Mozilla/5.0 (compatible; +centuryb.o.t9[at]gmail.com)
Mozilla/5.0 (compatible; +http://servernfo.com/)
Mozilla/5.0 (compatible; +http://tweetedtimes.com)
Mozilla/5.0 (compatible; +http://www.hosterstats.com/newgtld-websurvey.php)
Mozilla/5.0 (compatible; .None-Crawler/0.1 +http://domains.ericbinek.None/)
Mozilla/5.0 (compatible; 007ac9 Crawler; http://crawler.007ac9.net/)
Mozilla/5.0 (compatible; 008/0.83; http://www.80legs.com/spider.html;) Gecko/2008032620
Mozilla/5.0 (compatible; 008/0.83; http://www.80legs.com/webcrawler.html) Gecko/2008032620
Mozilla/5.0 (compatible; 008/0.83; http://www.80legs.com/webcrawler.html;) Gecko/2008032620
Mozilla/5.0 (compatible; 008/0.85; http://www.80legs.com/webcrawler.html) Gecko/2008032620
Mozilla/5.0 (compatible; 200PleaseBot/1.0; +http://www.200please.com/bot)
Mozilla/5.0 (compatible; 2GDPR/1.2; https://2gdpr.com)
Mozilla/5.0 (compatible; 4SeoHuntBot; +http://4seohunt.biz/about.html)
Mozilla/5.0 (compatible; 80bot/0.71; http://www.80legs.com/spider.html;) Gecko/2008032620
Mozilla/5.0 (compatible; ACHE/Unknown Version; +https://github.com/ViDA-NYU/ache; )
Mozilla/5.0 (compatible; ADJUSTbot/2.0; +http://www.ad-just.jp/)
Mozilla/5.0 (compatible; AMZNKAssocBot/4.0 +http://affiliate-program.amazon.com)
Mozilla/5.0 (compatible; AOLbot/4.0; +http://www.aol-soft.com/)
Mozilla/5.0 (compatible; AbiLogicBot/1.0; +http://www.abilogic.com/bot.html)
Mozilla/5.0 (compatible; Abonti/0.8 - http://www.abonti.com)
Mozilla/5.0 (compatible; Abonti/0.92 - http://www.abonti.com)
Mozilla/5.0 (compatible; AboutUsBot Johnny5/2.0; +http://www.AboutUs.org/)
Mozilla/5.0 (compatible; AboutUsBot/0.9; +http://www.aboutus.org/AboutUsBot)
Mozilla/5.0 (compatible; AcoonBot/4.10.6; +http://www.acoon.de/robot.asp)
Mozilla/5.0 (compatible; AcoonBot/4.10.7; +http://www.acoon.de/robot.asp)
Mozilla/5.0 (compatible; AcoonBot/4.10.8; +http://www.acoon.de/robot.asp)
Mozilla/5.0 (compatible; AcoonBot/4.11.0; +http://www.acoon.de/robot.asp)
Mozilla/5.0 (compatible; AcoonBot/4.11.1; +http://www.acoon.de/robot.asp)
Mozilla/5.0 (compatible; AcoonBot/4.12.1; +http://www.acoon.de/robot.asp)
Mozilla/5.0 (compatible; AddSearchBot/0.9; +http://www.addsearch.com/bot; info@addsearch.com)
Mozilla/5.0 (compatible; Adsbot/3.1;  https://seostar.co/robot/)
Mozilla/5.0 (compatible; AdvBot/2.0; +http://advbot.net/bot.html)
Mozilla/5.0 (compatible; AffiliateLabz/1.0; +https://affiliatelabz.com/)
Mozilla/5.0 (compatible; AhrefsBot/1.0; +http://ahrefs.com/robot/)
Mozilla/5.0 (compatible; AhrefsBot/2.0; +http://ahrefs.com/robot/)
Mozilla/5.0 (compatible; AhrefsBot/3.0; +http://ahrefs.com/robot/)
Mozilla/5.0 (compatible; AhrefsBot/3.1; +http://ahrefs.com/robot/
Mozilla/5.0 (compatible; AhrefsBot/3.1; +http://ahrefs.com/robot/)
Mozilla/5.0 (compatible; AhrefsBot/4.0; +http://ahrefs.com/robot/)
Mozilla/5.0 (compatible; AhrefsBot/5.0; +http://ahrefs.com/robot/)
Mozilla/5.0 (compatible; AhrefsBot/5.0; +http://ahrefs.com/robot/) AppEngine-Google; (+http://code.google.com/appengine; appid: s~proxyfile1-hrd)
Mozilla/5.0 (compatible; AhrefsBot/5.1; +http://ahrefs.com/robot/)
Mozilla/5.0 (compatible; AhrefsBot/5.2; +http://ahrefs.com/robot/)
Mozilla/5.0 (compatible; AhrefsBot/5.2; News; +http://ahrefs.com/robot/)
Mozilla/5.0 (compatible; AhrefsBot/6.1; +http://ahrefs.com/robot/)
Mozilla/5.0 (compatible; AhrefsBot/6.1; News; +http://ahrefs.com/robot/)
Mozilla/5.0 (compatible; AhrefsSiteAudit/5.2; +http://ahrefs.com/robot/)
Mozilla/5.0 (compatible; AhrefsSiteAudit/6.1; +http://ahrefs.com/robot/)
Mozilla/5.0 (compatible; AjaxSnapBot; +http://ajaxsnapshots.com/bot.html)
Mozilla/5.0 (compatible; Alexabot/1.0; +http://www.alexa.com/help/certifyscan; certifyscan@alexa.com)
Mozilla/5.0 (compatible; AlphaBot/3.2; +http://alphaseobot.com/bot.html)
Mozilla/5.0 (compatible; AndersPinkBot/1.0; +http://anderspink.com/bot.html)
Mozilla/5.0 (compatible; AntBot/1.0; +http://www.ant.com/)
Mozilla/5.0 (compatible; AnyApexBot/1.0; +http://www.anyapex.com/bot.html)
Mozilla/5.0 (compatible; Apercite; +http://www.apercite.fr/robot/index.html)
Mozilla/5.0 (compatible; AportWorm/3.2; +http://www.aport.ru/help)
Mozilla/5.0 (compatible; AppBeat/1.0; +http://www.appbeat.io/)
Mozilla/5.0 (compatible; Applebot/0.3; +http://www.apple.com/go/applebot)
Mozilla/5.0 (compatible; Aprc/2.9.15-24; +https://aprc.it/) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3528.4 Safari/537.36
Mozilla/5.0 (compatible; Arachnophilia/1.0; +http://arachnys.com/)
Mozilla/5.0 (compatible; Ask Jeeves/Teoma; +http://about.ask.com/en/docs/about/webmasters.shtml)
Mozilla/5.0 (compatible; AskPeterBot; +http://www.askpeter.de/bot.html)
Mozilla/5.0 (compatible; BIXOCRAWLER; +http://wiki.github.com/bixo/bixo/bixocrawler; bixo-dev@yahoogroups.com)
Mozilla/5.0 (compatible; BLEXBot/1.0; +http://webmeup-crawler.com/)
Mozilla/5.0 (compatible; BLEXBot/1.0; +http://webmeup.com/crawler.html)
Mozilla/5.0 (compatible; BLEXBotTest/1.0; +http://webmeup.com/crawler.html)
Mozilla/5.0 (compatible; Baiduspider-cpro; +http://www.baidu.com/search/spider.html)
Mozilla/5.0 (compatible; Baiduspider-render/2.0; +http://www.baidu.com/search/spider.html)
Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search
Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)
Mozilla/5.0 (compatible; BazQux/2.4; +https://bazqux.com/fetcher; 1 subscribers)
Mozilla/5.0 (compatible; BazQux/2.4; +https://bazqux.com/fetcher; 2 subscribers)
Mozilla/5.0 (compatible; BecomeBot/2.3; MSIE 6.0 compatible; +http://www.become.com/site_owners.html)
Mozilla/5.0 (compatible; BecomeBot/3.0; MSIE 6.0 compatible; +http://www.become.com/site_owners.html)
Mozilla/5.0 (compatible; BegunAdvertising/3.0; +http://begun.ru/begun/technology/indexer/)
Mozilla/5.0 (compatible; BehloolBot/beta; +http://www.webeaver.com/bot)
Mozilla/5.0 (compatible; Bender; http://sites.google.com/site/bendercrawler)
Mozilla/5.0 (compatible; BeslistBot; nl; BeslistBot 1.0; http://www.beslist.nl/
Mozilla/5.0 (compatible; BigBozz/2.2.1; +http://www.bigbozz.com/)
Mozilla/5.0 (compatible; BitBot/v1.19.0; +https://bitbot.dev)
Mozilla/5.0 (compatible; Blekkobot; ScoutJet; +http://blekko.com/about/blekkobot)
Mozilla/5.0 (compatible; BlinkaCrawler/1.0; +http://www.blinka.jp/crawler/)
Mozilla/5.0 (compatible; BlinkaCrawler/1.0; http://www.blinka.jp/crawler/)
Mozilla/5.0 (compatible; BlogTraffic/1.4 Feed-Fetcher; +http://www.blogtraffic.de/rss-bot.html)
Mozilla/5.0 (compatible; BomboraBot/1.0; +http://www.bombora.com/bot)
Mozilla/5.0 (compatible; BotPants/1.0; Linux; +info@copypants.com) KHTML/3.5.5 (like Gecko)
Mozilla/5.0 (compatible; BountiiBot/1.1; +http://bountii.com/contact.php)
Mozilla/5.0 (compatible; BoxcarBot/1.1; +awesome@boxcar.io)
Mozilla/5.0 (compatible; BusinessSeek.biz_Directory; http://www.businessseek.biz/)
Mozilla/5.0 (compatible; BusinessSeek.biz_Spider; http://www.businessseek.biz/)
Mozilla/5.0 (compatible; Butterfly/1.0; +http://labs.topsy.com/butterfly.html) Gecko/2009032608 Firefox/3.0.8
Mozilla/5.0 (compatible; Butterfly/1.0; +http://labs.topsy.com/butterfly/) Gecko/2009032608 Firefox/3.0.8
Mozilla/5.0 (compatible; BuzzSumo; +http://www.buzzsumo.com/bot.html)
Mozilla/5.0 (compatible; CareerBot/1.1; +http://www.career-x.de/bot.html)
Mozilla/5.0 (compatible; CensysInspect/1.1;  https://about.censys.io/)
Mozilla/5.0 (compatible; ChangesMeter/1.9.1; http://intuiware.com/apps/changes-meter)
Mozilla/5.0 (compatible; Charlotte/1.0b; charlotte@beta.spider.com)
Mozilla/5.0 (compatible; Charlotte/1.1; http://www.searchme.com/support/)
Mozilla/5.0 (compatible; Chirp/1.0; +http://www.binarycanary.com)
Mozilla/5.0 (compatible; Cincraw/1.0; +http://cincrawdata.net/bot/)
Mozilla/5.0 (compatible; CligooRobot/2.0; +http://www.cligoo.de/wk/technik.php)
Mozilla/5.0 (compatible; Cliqzbot/0.1 +http://cliqz.com/company/cliqzbot)
Mozilla/5.0 (compatible; Cliqzbot/1.0 +http://cliqz.com/company/cliqzbot)
Mozilla/5.0 (compatible; Cliqzbot/2.0; +http://cliqz.com/company/cliqzbot)
Mozilla/5.0 (compatible; CloudFlare-AlwaysOnline/1.0; +http://www.cloudflare.com/always-online)
Mozilla/5.0 (compatible; CloudFlare-AlwaysOnline/1.0; +http://www.cloudflare.com/always-online) AppleWebKit/534.34
Mozilla/5.0 (compatible; CloudFlare-AlwaysOnline/1.0; +https://www.cloudflare.com/always-online) AppleWebKit/534.34
Mozilla/5.0 (compatible; CloudServerMarketSpider/1.0; +http://cloudservermarket.com/spider.html)
Mozilla/5.0 (compatible; CloudServerMarketSpider/1.0; +http://www.cloudservermarket.com/spider.html)
Mozilla/5.0 (compatible; Cloudflare-AMP/1.0; +https://amp.cloudflare.com/doc/fetcher.html) AppleWebKit/534.34
Mozilla/5.0 (compatible; Cocolyzebot/1.0; https://cocolyze.com/bot)
Mozilla/5.0 (compatible; CompSpyBot/1.0; +http://www.compspy.com/spider.html)
Mozilla/5.0 (compatible; CrawlBot/1.0.0)
Mozilla/5.0 (compatible; CrawlRobot; +http://localhost/bot.php)
Mozilla/5.0 (compatible; Crawler/0.9; http://linkfluence.net/)
Mozilla/5.0 (compatible; Crawlera/1.10.2; UID 11912)
Mozilla/5.0 (compatible; Crawlera/1.10.2; UID 24522)
Mozilla/5.0 (compatible; Crawlera/1.10.2; UID 40409)
Mozilla/5.0 (compatible; Crawlera/1.10.2; UID 43063)
Mozilla/5.0 (compatible; Crawlera/1.10.2; UID 47129)
Mozilla/5.0 (compatible; Crawlera/1.10.2; UID 52327)
Mozilla/5.0 (compatible; Crawlera/1.10.2; UID 6573)
Mozilla/5.0 (compatible; Crawlera/1.10.2; UID 70054)
Mozilla/5.0 (compatible; Crawlera/1.10.2; UID 70350)
Mozilla/5.0 (compatible; Crawlera/1.10.2; UID 73393)
Mozilla/5.0 (compatible; Crawlera/1.10.2; UID 75425)
Mozilla/5.0 (compatible; Crawlera/1.10.2; UID 84846)
Mozilla/5.0 (compatible; Crawlera/1.10.2; UID 85747)
Mozilla/5.0 (compatible; Crawlera/1.10.2; UID 87214)
Mozilla/5.0 (compatible; Crawlera/1.10.2; UID 8942)
Mozilla/5.0 (compatible; Crawlera/1.10.2; UID 92751)
Mozilla/5.0 (compatible; Crawly/1.9; http://92.51.162.40/crawler.html)
Mozilla/5.0 (compatible; CukBot; Not a spammer; http://www.companiesintheuk.co.uk/bot.html)
Mozilla/5.0 (compatible; DBLBot/1.0; +http://www.dontbuylists.com/)
Mozilla/5.0 (compatible; DCPbot/1.0; +http://domains.checkparams.com/)
Mozilla/5.0 (compatible; DCPbot/1.1; +http://domains.checkparams.com/)
Mozilla/5.0 (compatible; DCPbot/1.2; +http://domains.checkparams.com/)
Mozilla/5.0 (compatible; DCPbot/1.4; +http://domains.checkparams.com/)
Mozilla/5.0 (compatible; DCPbot/1.5; +http://domains.checkparams.com/)
Mozilla/5.0 (compatible; DIY-SEOBot/0.1a; +http://www.upcity.com/bot.html)
Mozilla/5.0 (compatible; DKIMRepBot/1.0; +http://www.dkim-reputation.org)
Mozilla/5.0 (compatible; DNS SSL/TLS HTTP HTML Website Security Scanner/0.2 beta; +https://www.htmlyse.com/)
Mozilla/5.0 (compatible; DNS SSL/TLS HTTP and HTML scanner/0.1; +https://www.htmlyse.com/)
Mozilla/5.0 (compatible; DNS-Digger-Explorer/1.0; +http://www.dnsdigger.com)
Mozilla/5.0 (compatible; DNS-Digger/1.0; +http://www.dnsdigger.com)
Mozilla/5.0 (compatible; DNSResearchBot/2.1; +http://195.37.190.77)
Mozilla/5.0 (compatible; DataForSeoBot/1.0; +https://dataforseo.com/dataforseo-bot)
Mozilla/5.0 (compatible; DataXu/1.0; +http://dataxu.com)
Mozilla/5.0 (compatible; Dataprovider Site Explorer; http://www.dataprovider.com/)
Mozilla/5.0 (compatible; Dataprovider.com)
Mozilla/5.0 (compatible; Dataprovider.com;)
Mozilla/5.0 (compatible; Dataprovider/6.101; +https://www.dataprovider.com/)
Mozilla/5.0 (compatible; Dataprovider/6.92; +https://www.dataprovider.com/)
Mozilla/5.0 (compatible; Dataprovider; https://www.dataprovider.com/spider/)
Mozilla/5.0 (compatible; Daum/4.1; +http://cs.daum.net/faq/15/4118.html?faqId=28966)
Mozilla/5.0 (compatible; Dazoobot/0.1; +http://dazoo.fr)
Mozilla/5.0 (compatible; Dazoobot/1.0; http://dazoo.fr)
Mozilla/5.0 (compatible; Dead Link Checker; http://www.dead-link-checker.com/)
Mozilla/5.0 (compatible; DeuSu/0.1.0; +https://deusu.de/robot.html)
Mozilla/5.0 (compatible; DeuSu/0.1.0; +https://deusu.org)
Mozilla/5.0 (compatible; DeuSu/5.0.2; +https://deusu.de/robot.html)
Mozilla/5.0 (compatible; Diffbot/0.1; +http://www.diffbot.com)
Mozilla/5.0 (compatible; Digincore bot; https://www.digincore.com/crawler.html for rules and instructions.
Mozilla/5.0 (compatible; Digincore bot; https://www.digincore.com/crawler.html for rules and instructions.)
Mozilla/5.0 (compatible; Discordbot/1.0; +https://discordapp.com)
Mozilla/5.0 (compatible; Discordbot/2.0; +https://discordapp.com)
Mozilla/5.0 (compatible; DnyzBot/1.0)
Mozilla/5.0 (compatible; DomainAppender /1.0; +http://www.profound.net/domainappender)
Mozilla/5.0 (compatible; DomainMacroCrawler/0.1; +http://domainmacro.com)
Mozilla/5.0 (compatible; DomainSONOCrawler/0.1; +http://domainsono.com)
Mozilla/5.0 (compatible; DomainSigmaCrawler/0.1; +http://domainsigma.com/robot)
Mozilla/5.0 (compatible; DomainTunoCrawler/0.1; +http://www.domaintuno.com/robot)
Mozilla/5.0 (compatible; DomainTunoCrawler/0.1; +https://www.domaintuno.com/robot)
Mozilla/5.0 (compatible; Domains Project/1.0.3; +https://github.com/tb0hdan/domains)
Mozilla/5.0 (compatible; Domains Project/1.0.8; +https://domainsproject.org)
Mozilla/5.0 (compatible; Domains Project/1.1.0; +https://domainsproject.org)
Mozilla/5.0 (compatible; DotBot/1.1; http://www.dotnetdotcom.org/, crawler@dotnetdotcom.org)
Mozilla/5.0 (compatible; DotBot/1.1; http://www.opensiteexplorer.org/dotbot, help@moz.com)
Mozilla/5.0 (compatible; Dow Jones Searchbot)
Mozilla/5.0 (compatible; DripfeedBot/2.0; +http://dripfeedbookmark.com/bot.html
Mozilla/5.0 (compatible; Dubbotbot/0.2; +http://dubbot.com)
Mozilla/5.0 (compatible; DuckDuckBot-Https/1.1; https://duckduckgo.com/duckduckbot)
Mozilla/5.0 (compatible; DuckDuckGo-Favicons-Bot/1.0; +http://duckduckgo.com)
Mozilla/5.0 (compatible; EARTHCOM/2.2; +http://enter4u.eu)
Mozilla/5.0 (compatible; EasouSpider; +http://www.easou.com/search/spider.html)
Mozilla/5.0 (compatible; Embedly/0.2; +http://support.embed.ly/)
Mozilla/5.0 (compatible; Embedly/0.2; snap; +http://support.embed.ly/)
Mozilla/5.0 (compatible; Esribot/1.0; http://www.esrihu.hu/)
Mozilla/5.0 (compatible; EtaoSpider/1.0; http://open.etao.com/dev/EtaoSpider)
Mozilla/5.0 (compatible; EuripBot/2.0; +http://www.eurip.com)
Mozilla/5.0 (compatible; EventGuruBot/1.0; +http://www.eventguru.com/spider.html)
Mozilla/5.0 (compatible; EveryoneSocialBot/1.0; support@everyonesocial.com http://everyonesocial.com/)
Mozilla/5.0 (compatible; Evrinid Iudex 1.0.0; +http://www.evri.com/evrinid)
Mozilla/5.0 (compatible; Ex-Crawler/0.1.5a; powered by ex-crawler; +http://www.ex-crawler.de/) Java/1.6.0_20,gzip(gfe),gzip(gfe)
Mozilla/5.0 (compatible; Exabot PyExalead/3.0; +http://www.exabot.com/go/robot)
Mozilla/5.0 (compatible; Exabot-Images/3.0; +http://www.exabot.com/go/robot)
Mozilla/5.0 (compatible; Exabot/3.0 (BiggerBetter); +http://www.exabot.com/go/robot)
Mozilla/5.0 (compatible; Exabot/3.0;  http://www.exabot.com/go/robot)
Mozilla/5.0 (compatible; Exabot/3.0; +http://www.exabot.com/go/robot)
Mozilla/5.0 (compatible; ExchangleBot/3.0; +https://www.exchangle.com/exchangling)
Mozilla/5.0 (compatible; ExpertSearchSpider +http://www.expertsearch.nl/spider)
Mozilla/5.0 (compatible; Exploratodo/1.0; +http://www.exploratodo.com
Mozilla/5.0 (compatible; ExtLinksBot/1.5 +https://extlinks.com/Bot.html)
Mozilla/5.0 (compatible; Eyeotabot/1.0; +http://www.eyeota.com)
Mozilla/5.0 (compatible; Ezooms/1.0; ezooms.bot@gmail.com)
Mozilla/5.0 (compatible; Ezooms/1.0; help@moz.com)
Mozilla/5.0 (compatible; FAST Crawler 6.3)
Mozilla/5.0 (compatible; Falconsbot; +http://iws.seu.edu.cn/services/falcons/contact_us.jsp)
Mozilla/5.0 (compatible; Falconsbot; +http://ws.nju.edu.cn/falcons/)
Mozilla/5.0 (compatible; FatBot 2.0; http://www.thefind.com/crawler)
Mozilla/5.0 (compatible; Faveeo/1.0; +http://www.faveeo.com)
Mozilla/5.0 (compatible; FeedBooster; +http://feeds.qsensei.com)
Mozilla/5.0 (compatible; Feedage/2.0; +http://www.feedage.com/bot.php)
Mozilla/5.0 (compatible; Feedspot/1.0 (+https://www.feedspot.com/fs/fetcher; like FeedFetcher-Google)
Mozilla/5.0 (compatible; Feedspotbot/1.0; +http://www.feedspot.com/fs/bot)
Mozilla/5.0 (compatible; Feedwind/2.0; +http://feed.mikle.com/support/description/)
Mozilla/5.0 (compatible; Feedwind/3.0; +http://feed.mikle.com/support/description/)
Mozilla/5.0 (compatible; FemtosearchBot/1.0; http://femtosearch.com)
Mozilla/5.0 (compatible; Finderbots finder bot; +http://wiki.github.com/bixo/bixo/bixocrawler; bixo-dev@yahoogroups.com)
Mozilla/5.0 (compatible; Findxbot/1.0; +http://www.findxbot.com)
Mozilla/5.0 (compatible; Firefox compatible; MS IE compatible; http://search.daum.net/) Daumoa-feedfetcher/2.0
Mozilla/5.0 (compatible; FlipboardProxy/1.1; +http://flipboard.com/browserproxy)
Mozilla/5.0 (compatible; FlipboardProxy/1.2; +http://flipboard.com/browserproxy)
Mozilla/5.0 (compatible; FlipboardRSS/1.1; +http://flipboard.com/browserproxy)
Mozilla/5.0 (compatible; FlipboardRSS/1.2; +http://flipboard.com/browserproxy)
Mozilla/5.0 (compatible; FoundSeoTool/1.0; +http://www.found.co.uk/seo-tool/)
Mozilla/5.0 (compatible; FunkyCrawler; )
Mozilla/5.0 (compatible; Funnelback) RPT-HTTPClient/0.3-3E
Mozilla/5.0 (compatible; GeliyooBot/1.0; +http://www.geliyoo.com/)
Mozilla/5.0 (compatible; GeliyooBot/1.0beta; +http://www.geliyoo.com/)
Mozilla/5.0 (compatible; Genieo/1.0 http://www.genieo.com/webfilter.html)
Mozilla/5.0 (compatible; Genieo/1.0 http://www.genieo.com/webfilter.html) AppEngine-Google; (+http://code.google.com/appengine; appid: s~natmacdevice)
Mozilla/5.0 (compatible; GigaBot/1.0; +http://www.gigablast.com/ )
Mozilla/5.0 (compatible; GigablastOpenSource/1.0)
Mozilla/5.0 (compatible; Gimme60bot/1.0; +http://gimme60.com)
Mozilla/5.0 (compatible; Gimme60bot/1.0; +http://gimme60.com) Firefox/16.0
Mozilla/5.0 (compatible; GimmeUSAbot/1.0; +http://gimmeusa-update.com/crawler)
Mozilla/5.0 (compatible; GimmeUSAbot/1.0; +http://gimmeusa.com/crawler)
Mozilla/5.0 (compatible; GimmeUSAbot/1.0; +http://gimmeusa.com/crawler.html)
Mozilla/5.0 (compatible; GimmeUSAbot/1.0; +https://gimmeusa.com/crawler.html)
Mozilla/5.0 (compatible; GimmeUSAbot/1.0; +https://gimmeusa.com/pages/crawler)
Mozilla/5.0 (compatible; Gluten Free Crawler/1.0; +http://glutenfreepleasure.com/)
Mozilla/5.0 (compatible; Go-http-client/1.1; +centurybot9@gmail.com)
Mozilla/5.0 (compatible; Goodzer/2.0; crawler@goodzer.com)
Mozilla/5.0 (compatible; Google AppsViewer; http://drive.google.com)
Mozilla/5.0 (compatible; Google Desktop/5.9.1005.12335; http://desktop.google.com/)
Mozilla/5.0 (compatible; Google Keyword Tool; +http://adwords.google.com/select/KeywordToolExternal)
Mozilla/5.0 (compatible; Google-Apps-Script)
Mozilla/5.0 (compatible; Google-Podcast)
Mozilla/5.0 (compatible; Google-Site-Verification/1.0)
Mozilla/5.0 (compatible; Google-Structured-Data-Testing-Tool +http://developers.google.com/structured-data/testing-tool/)
Mozilla/5.0 (compatible; Google-Structured-Data-Testing-Tool +https://search.google.com/structured-data/testing-tool)
Mozilla/5.0 (compatible; Google-Youtube-Links)
Mozilla/5.0 (compatible; GoogleDocs; apps-spreadsheets; +http://docs.google.com)
Mozilla/5.0 (compatible; Googlebot/2.1; +http://import.io)
Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
Mozilla/5.0 (compatible; Googlebot/2.1; https://deepcrawl.com/bot)
Mozilla/5.0 (compatible; Googlebot/2.1; https://www.deepcrawl.com/bot)
Mozilla/5.0 (compatible; Googlebot/2.1; startmebot/1.0; +https://start.me/bot)
Mozilla/5.0 (compatible; Gookey.co/1.0; +http://gookey.co/)
Mozilla/5.0 (compatible; Gowikibot/1.0; +http://www.gowikibot.com)
Mozilla/5.0 (compatible; GrapeshotCrawler/2.0; +http://www.grapeshot.co.uk/crawler.php)
Mozilla/5.0 (compatible; GrapeshotCrawler/2.0; +https://www.grapeshot.com/crawler/)
Mozilla/5.0 (compatible; Grobbot/2.2; +https://grob.it)
Mozilla/5.0 (compatible; GroupHigh/1.0; +http://www.grouphigh.com/
Mozilla/5.0 (compatible; GroupHigh/1.0; +http://www.grouphigh.com/)
Mozilla/5.0 (compatible; GroupHigh/1.0; +http://www.grouphigh.com/bot.html)
Mozilla/5.0 (compatible; GurujiBot/1.0; +http://www.guruji.com/en/WebmasterFAQ.html)
Mozilla/5.0 (compatible; Hailoobot/1.2; +http://www.hailoo.com/spider.html)
Mozilla/5.0 (compatible; Hailoobot/1.2; http://www.hailoo.com/spider.html)
Mozilla/5.0 (compatible; Heurekabot/3.1; +http://sluzby.heureka.cz/)
Mozilla/5.0 (compatible; HiveCrawler/1.2 http://www.businessinsider.com)
Mozilla/5.0 (compatible; HomeTags/1.0;  http://www.hometags.nl/bot)
Mozilla/5.0 (compatible; HomeTags/1.0; +http://www.hometags.nl/bot)
Mozilla/5.0 (compatible; Host-Spy Crawler/1.2; +http://www.host-spy.com/)
Mozilla/5.0 (compatible; HubSpot Crawler; web-crawlers@hubspot.com)
Mozilla/5.0 (compatible; HyperCrawl/0.2; +http://www.seograph.net/bot.html)
Mozilla/5.0 (compatible; ICBot/0.1; +https://ideasandcode.xyz
Mozilla/5.0 (compatible; IODC-Internet Engineering Odysseus 001; +http://iodc.co.uk)
Mozilla/5.0 (compatible; IODC-Internet Engineering Odysseus 005; +http://iodc.co.uk)
Mozilla/5.0 (compatible; IODC-Internet Engineering Odysseus 006; +http://iodc.co.uk)
Mozilla/5.0 (compatible; IODC-Internet Engineering Odysseus 007; +http://iodc.co.uk)
Mozilla/5.0 (compatible; IODC-Internet Engineering Odysseus 011; +http://iodc.co.uk)
Mozilla/5.0 (compatible; IODC-Internet Engineering Odysseus 013; +http://iodc.co.uk)
Mozilla/5.0 (compatible; IODC-Odysseus Survey 29691-100-070315021100-72; +http://iodc.co.uk)
Mozilla/5.0 (compatible; IODC-Odysseus Survey 29692-100-060415034231-192; +http://iodc.co.uk)
Mozilla/5.0 (compatible; IODC-Odysseus Survey 31193-100-080615161452-372; +http://iodc.co.uk)
Mozilla/5.0 (compatible; IODC-Odysseus Survey 37044-100-110516191742-025; +https://iodc.co.uk)
Mozilla/5.0 (compatible; IODC-Odysseus Survey 41437-100-090715214314-122; +http://iodc.co.uk)
Mozilla/5.0 (compatible; IXEbot; +http://medialab.di.unipi.it/IXEbot.html)
Mozilla/5.0 (compatible; ImageFetcher/5.6; +http://images.weserv.nl/)
Mozilla/5.0 (compatible; InbyBot/2.1; +http://inbyapp.com)
Mozilla/5.0 (compatible; Infohelfer/1.2.0; +http://www.infohelfer.de/crawler.php)
Mozilla/5.0 (compatible; Infohelfer/1.3.0; +http://www.infohelfer.de/crawler.php)
Mozilla/5.0 (compatible; Infohelfer/1.3.3; +http://www.infohelfer.de/crawler.php)
Mozilla/5.0 (compatible; Infohelfer/1.4.3; +http://www.infohelfer.de/crawler.php)
Mozilla/5.0 (compatible; InfuzApp/1.0; +http://www.infuz.com/bot.html)
Mozilla/5.0 (compatible; IstellaBot/1.01.18 +http://www.tiscali.it/)
Mozilla/5.0 (compatible; IstellaBot/1.10.2 +http://www.tiscali.it/)
Mozilla/5.0 (compatible; IstellaBot/1.18.81 +http://www.tiscali.it/)
Mozilla/5.0 (compatible; IstellaBot/1.23.15 +http://www.tiscali.it/)
Mozilla/5.0 (compatible; JadynAveBot; +http://www.jadynave.com/robot)
Mozilla/5.0 (compatible; Javelin; +https://about.javelin.io/)
Mozilla/5.0 (compatible; Jetslide; +http://jetsli.de/crawler)
Mozilla/5.0 (compatible; JikeSpider; +http://shoulu.jike.com/spider.html)
Mozilla/5.0 (compatible; JobKereso;  +http://www.kozvetlen-allasok.hu/robot.jsp  info@kozvetlen-allasok.hu)
Mozilla/5.0 (compatible; JobKereso;  +https://www.kozvetlen-allasok.hu/robot.jsp  info@kozvetlen-allasok.hu)
Mozilla/5.0 (compatible; JobKeresoBot; +http://www.kozvetlen-allasok.hu/help.jsp; info@kozvetlen-allasok.hu; 5.0
Mozilla/5.0 (compatible; JobKeresoBot; +https://www.kozvetlen-allasok.hu/help.jsp; info@kozvetlen-allasok.hu; 5.0
Mozilla/5.0 (compatible; JobdiggerSpider +http://www.jobdigger.nl/spider)
Mozilla/5.0 (compatible; Jobrapido/1.1; +http://www.jobrapido.com)
Mozilla/5.0 (compatible; Jooblebot/2.0; Windows NT 6.1; WOW64; +http://jooble.org/jooble-bot) AppleWebKit/537.36 (KHTML, like Gecko) Safari/537.36
Mozilla/5.0 (compatible; Jooblebot/2.0; Windows NT 6.1; WOW64; +http://jooble.org/jooble-bot) Mobile
Mozilla/5.0 (compatible; JungleKeyThumbnail/1.1; +http://www.junglekey.fr/)
Mozilla/5.0 (compatible; KAZ.KZ_Bot/3.0)
Mozilla/5.0 (compatible; KHTML, like Gecko) Chrome/43.0.2357.81 Safari/537.36 collection@infegy.com
Mozilla/5.0 (compatible; KaloogaBot; http://kalooga.com/crawler)
Mozilla/5.0 (compatible; KaloogaBot; http://www.kalooga.com/info.html?page=crawler)
Mozilla/5.0 (compatible; Kelny/2.0; +http://www.kelny.com/)
Mozilla/5.0 (compatible; Keywords Research; http://www.check-domains.com/keyword-research/keywords-adwords.php)
Mozilla/5.0 (compatible; KoepaBot BETA; http://www.koepa.nl/bot.html)
Mozilla/5.0 (compatible; Konqueror/3.5; Linux) KHTML/3.5.5 (like Gecko) (Exabot-Thumbnails)
Mozilla/5.0 (compatible; Kraken/0.1; http://linkfluence.net/; bot@linkfluence.net)
Mozilla/5.0 (compatible; Kyluka crawl; crawl@kyluka.com; http://www.kyluka.com/static/crawl.html)
Mozilla/5.0 (compatible; LA1; +http://www.zeerch.com/bot.php)
Mozilla/5.0 (compatible; LA2; +http://www.zeerch.com/zeerch2/bot.php)
Mozilla/5.0 (compatible; LXRbot/1.0; http://lxrseo.com/, support@lxrseo.com)
Mozilla/5.0 (compatible; LXRbot/1.0;http://www.lxrmarketplace.com/,support@lxrmarketplace.com)
Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)
Mozilla/5.0 (compatible; Licorne Image Snapshot 1.0; +http://www.jednorozec.cz)
Mozilla/5.0 (compatible; LikaholixCrawler/1.0; +http://mylikes.com/about/crawler)
Mozilla/5.0 (compatible; LinkMarketbot/1.2; +http://www.linkmarket.com/)
Mozilla/5.0 (compatible; LinkisBot/1.0; bot@linkis.com) (iPhone; CPU iPhone OS 8_4_1 like Mac OS X) Mobile/12H321
Mozilla/5.0 (compatible; LinkpadBot/1.06; +http://www.linkpad.ru)
Mozilla/5.0 (compatible; LinkpadBot/1.07; +http://www.linkpad.ru)
Mozilla/5.0 (compatible; LinkpadBot/1.08; +http://www.linkpad.ru)
Mozilla/5.0 (compatible; LinkpadBot/2.3; +http://linkpad.org/robot/)
Mozilla/5.0 (compatible; LinksManager.com_bot +http://linksmanager.com/linkchecker.html)
Mozilla/5.0 (compatible; LinksManager.com_bot http://linksmanager.com/linkchecker.html)
Mozilla/5.0 (compatible; Linux i686; Yandex.Gazeta Bot/1.0; +http://gazeta.yandex.ru)
Mozilla/5.0 (compatible; Linux i686; en-US; URLfilterDB-crawler/1.0) ufdb/1.0
Mozilla/5.0 (compatible; Linux x86_64; Mail.RU_Bot/2.0; +http://go.mail.ru/
Mozilla/5.0 (compatible; Linux x86_64; Mail.RU_Bot/2.0; +http://go.mail.ru/help/robots)
Mozilla/5.0 (compatible; Linux x86_64; Mail.RU_Bot/Fast/2.0; +http://go.mail.ru/help/robots)
Mozilla/5.0 (compatible; Linux x86_64; Mail.RU_Bot/Img/2.0; +http://go.mail.ru/help/robots)
Mozilla/5.0 (compatible; Linux x86_64; Mail.RU_Bot/Robots/2.0; +http://go.mail.ru/help/robots)
Mozilla/5.0 (compatible; Linux x86_64; Mail.RU_Bot/Robots; +http://go.mail.ru/help/robots)
Mozilla/5.0 (compatible; Linux; InfegyAtlas/1.0; en-US; collection@infegy.com)
Mozilla/5.0 (compatible; Linux; Socialradarbot/2.0; en-US; bot@infegy.com)
Mozilla/5.0 (compatible; Linux; Socialradarbot/2.0; en-US; crawler@infegy.com)
Mozilla/5.0 (compatible; Lipperhey Link Explorer; http://www.lipperhey.com/)
Mozilla/5.0 (compatible; Lipperhey SEO Service; http://www.lipperhey.com/)
Mozilla/5.0 (compatible; Lipperhey Site Explorer; http://www.lipperhey.com/)
Mozilla/5.0 (compatible; Lipperhey Spider; http://www.lipperhey.com/)
Mozilla/5.0 (compatible; Lipperhey-Kaus-Australis/5.0; +https://www.lipperhey.com/en/about/)
Mozilla/5.0 (compatible; Lippershey/5.2; +https://www.lipperhey.com/en/about/)
Mozilla/5.0 (compatible; LoadTimeBot/0.7; +http://www.load-time.com/bot.html)
Mozilla/5.0 (compatible; LoadTimeBot/0.81; +http://www.load-time.com/bot.html)
Mozilla/5.0 (compatible; LoadTimeBot/0.9; +http://www.loadtime.net/bot.html)
Mozilla/5.0 (compatible; LucidWorks/; +; crawler at example dot com)
Mozilla/5.0 (compatible; LuminateBot/1.0; +http://www.luminate.com/bot/)
Mozilla/5.0 (compatible; LumtelBot/1.0; +http://lumtel.com)
Mozilla/5.0 (compatible; METASpider; +http://meta.ua/spider)
Mozilla/5.0 (compatible; MFGPagesBot/2.1; http://www.mfgpages.com)
Mozilla/5.0 (compatible; MJ12bot/v1.2.1; http://www.majestic12.co.uk/bot.php?+)
Mozilla/5.0 (compatible; MJ12bot/v1.2.3; http://www.majestic12.co.uk/bot.php?+)
Mozilla/5.0 (compatible; MJ12bot/v1.2.4; http://www.majestic12.co.uk/bot.php?+)
Mozilla/5.0 (compatible; MJ12bot/v1.2.5; http://www.majestic12.co.uk/bot.php?+)
Mozilla/5.0 (compatible; MJ12bot/v1.3.0; http://www.majestic12.co.uk/bot.php?+)
Mozilla/5.0 (compatible; MJ12bot/v1.3.1; http://www.majestic12.co.uk/bot.php?+)
Mozilla/5.0 (compatible; MJ12bot/v1.3.2; http://www.majestic12.co.uk/bot.php?+)
Mozilla/5.0 (compatible; MJ12bot/v1.3.3; http://www.majestic12.co.uk/bot.php?+)
Mozilla/5.0 (compatible; MJ12bot/v1.4.0; http://www.majestic12.co.uk/bot.php?+)
Mozilla/5.0 (compatible; MJ12bot/v1.4.1; http://www.majestic12.co.uk/bot.php?+)
Mozilla/5.0 (compatible; MJ12bot/v1.4.2; http://www.majestic12.co.uk/bot.php?+)
Mozilla/5.0 (compatible; MJ12bot/v1.4.3; http://www.majestic12.co.uk/bot.php?+)
Mozilla/5.0 (compatible; MJ12bot/v1.4.4 (domain ownership verifier); http://www.majestic12.co.uk/bot.php?+)
Mozilla/5.0 (compatible; MJ12bot/v1.4.4; http://www.majestic12.co.uk/bot.php?+)
Mozilla/5.0 (compatible; MJ12bot/v1.4.5; http://www.majestic12.co.uk/bot.php?+)
Mozilla/5.0 (compatible; MJ12bot/v1.4.6; http://mj12bot.com/)
Mozilla/5.0 (compatible; MJ12bot/v1.4.7; http://mj12bot.com/)
Mozilla/5.0 (compatible; MJ12bot/v1.4.7; http://www.majestic12.co.uk/bot.php?+)
Mozilla/5.0 (compatible; MJ12bot/v1.4.8; http://mj12bot.com/)
Mozilla/5.0 (compatible; MOSBookmarks/v2.6-Plus; Link Checker)
Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.1; Trident/6.0) LinkCheck by Siteimprove.com
Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.1; Trident/6.0) PTST/276
Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.1; Trident/6.0) SiteCheck-sitecrawl by Siteimprove.com
Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.1; Trident/6.0; .NET CLR 1.1.4322; .NET CLR 1.0.3705; .NET CLR 2.0.50727; .NET CLR 3.0.04506.648; .NET CLR 3.5.21022; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729) DMBrowser/2.1
Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.1; WOW64; Trident/6.0) PTST/276
Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.1; WOW64; Trident/6.0) PTST/277
Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.1; WOW64; Trident/6.0) PTST/281
Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.1; WOW64; Trident/6.0) PTST/284
Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.2; Trident/6.0;  WOW64;  Trident/6.0;  BingPreview/1.0b)
Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.2; Trident/6.0; WOW64; Trident/6.0; BingPreview/1.0b)
Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.2; WOW64; Trident/6.0) CrawlerProcess (http://www.PowerMapper.com) /5.10.736.0
Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.2; WOW64; Trident/6.0) CrawlerProcess (http://www.PowerMapper.com) /5.11.740.0
Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.2; WOW64; Trident/6.0) CrawlerProcess (http://www.PowerMapper.com) /5.21.754.0
Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.2; WOW64; Trident/6.0) CrawlerProcess (http://www.PowerMapper.com) /5.23.770.0
Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.2; WOW64; Trident/6.0) CrawlerProcess (http://www.PowerMapper.com) /5.24.776.0
Mozilla/5.0 (compatible; MSIE 7.0 +http://www.europarchive.org)
Mozilla/5.0 (compatible; MSIE 8.0; Windows NT 5.1) KomodiaBot/1.0
Mozilla/5.0 (compatible; MSIE 8.0; Windows NT 5.1) Streamline3Bot/1.0
Mozilla/5.0 (compatible; MSIE 9.0; Firefox/50.1) Daum/4.1
Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; CrawlDaddy v0.3.0 abot v1.1.1.0 http://code.google.com/p/abot)
Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Multiviewbot
Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/5.0) PTST/276
Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/5.0); 360Spider
Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/5.0); 360Spider(compatible; HaosouSpider; http://www.haosou.com/help/help_3_2.html)
Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/5.0;  WOW64;  Trident/5.0;  BingPreview/1.0b)
Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/5.0; AppInsights)
Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/5.0; CHKD 1.2; Akamai_Site_Analyzer)
Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/5.0; PTST 2.385)
Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; WOW64; Trident/5.0) commoncrawl.org/research//Nutch-1.7-SNAPSHOT
Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; WOW64; Trident/5.0; PTST 2.386)
Mozilla/5.0 (compatible; MSIE 9.0; Windows NT; MarketwireBot +http://www.marketwire.com)
Mozilla/5.0 (compatible; MSIE or Firefox mutant; not on Windows server; + http://tab.search.daum.net/aboutWebSearch.html) Daumoa/3.0
Mozilla/5.0 (compatible; MSIE or Firefox mutant; not on Windows server; +http://ws.daum.net/aboutWebSearch.html) Daumoa/2.0
Mozilla/5.0 (compatible; MSIE or Firefox mutant; not on Windows server; +http://ws.daum.net/aboutWebSearch.html) Daumoa/3.0
Mozilla/5.0 (compatible; MSIE or Firefox mutant; not on Windows server;) Daum 4.1
Mozilla/5.0 (compatible; MSIE or Firefox mutant; not on Windows server;) Daumoa 4.0
Mozilla/5.0 (compatible; MSIE or Firefox mutant; not on Windows server;) Daumoa/4.0
Mozilla/5.0 (compatible; MSIE or Firefox mutant; not on Windows server;) Daumoa/4.0 (Following Mediapartners-Google)
Mozilla/5.0 (compatible; MSIE or Firefox mutant; not on Windows server;) Daumoa/4.0;
Mozilla/5.0 (compatible; MSIE or Firefox mutant;) Daum 4.1
Mozilla/5.0 (compatible; MagiBot/3.4.3; +http://magi.peak-labs.com/robots.txt)
Mozilla/5.0 (compatible; MagiBot/3.6.2; +http://magi.peak-labs.com/robots.txt)
Mozilla/5.0 (compatible; Magibot/4.0.0; +http://magi.peak-labs.com/robots.txt)
Mozilla/5.0 (compatible; Mail.RU_Bot/2.0)
Mozilla/5.0 (compatible; Mail.RU_Bot/2.0; +http://go.mail.ru/
Mozilla/5.0 (compatible; Mail.RU_Bot/2.0; +http://go.mail.ru/help/robots)
Mozilla/5.0 (compatible; Mappy/1.0; +http://mappydata.net/bot/)
Mozilla/5.0 (compatible; MegaIndex.com/2.0; +http://megaindex.com/crawler)
Mozilla/5.0 (compatible; MegaIndex.ru/2.0; +http://megaindex.com/crawler)
Mozilla/5.0 (compatible; MegaIndex.ru/2.0; +https://www.megaindex.ru/?tab=linkAnalyze)
Mozilla/5.0 (compatible; MerchantCentricBot/1.0)
Mozilla/5.0 (compatible; MetaJobBot; http://www.metajob.at/crawler)
Mozilla/5.0 (compatible; MetaJobBot; http://www.metajob.de/crawler)
Mozilla/5.0 (compatible; MetamojiCrawler/1.0; +http://www.metamoji.com/jp/crawler.html
Mozilla/5.0 (compatible; Miniflux/2.0.10; +https://miniflux.net)
Mozilla/5.0 (compatible; Miniflux/2.0.11; +https://miniflux.app)
Mozilla/5.0 (compatible; Miniflux/2.0.12; +https://miniflux.app)
Mozilla/5.0 (compatible; Miniflux/2.0.3; +https://miniflux.net)
Mozilla/5.0 (compatible; Miniflux/2.0.7; +https://miniflux.net)
Mozilla/5.0 (compatible; Miniflux/2.0.x-dev; +https://miniflux.net)
Mozilla/5.0 (compatible; Miniflux/3b6e44c; +https://miniflux.app)
Mozilla/5.0 (compatible; Miniflux/ae1dc1a; +https://miniflux.app)
Mozilla/5.0 (compatible; MixrankBot; crawler@mixrank.com)
Mozilla/5.0 (compatible; MojeekBot/0.2; http://www.mojeek.com/bot.html#relaunch)
Mozilla/5.0 (compatible; MojeekBot/0.2; http://www.mojeek.com/bot.html)
Mozilla/5.0 (compatible; MojeekBot/0.5; http://www.mojeek.com/bot.html)
Mozilla/5.0 (compatible; MojeekBot/0.6; +https://www.mojeek.com/bot.html)
Mozilla/5.0 (compatible; MojeekBot/0.6; http://www.mojeek.com/bot.html)
Mozilla/5.0 (compatible; MotoMinerBot/1.0; +https://motominer.com/Bot)
Mozilla/5.0 (compatible; Mozilla/5.0; +http://wiki.github.com/bixo/bixo/bixocrawler; bixo-dev@yahoogroups.com)
Mozilla/5.0 (compatible; Mozilla; +http://wiki.github.com/bixo/bixo/bixocrawler; bixo-dev@yahoogroups.com)
Mozilla/5.0 (compatible; Mp3Bot/0.4; +http://mp3realm.org/mp3bot/)
Mozilla/5.0 (compatible; Mp3Bot/0.7; +http://mp3realm.org/mp3bot/)
Mozilla/5.0 (compatible; MuckRack/1.0; +http://muckrack.com)
Mozilla/5.0 (compatible; Muenster University of Applied Sciences; +http://fb02itsscan.fh-muenster.de)
Mozilla/5.0 (compatible; NIXStatsbot/1.1; +http://www.nixstats.com/bot.html)
Mozilla/5.0 (compatible; NLCrawler/2.0.15; Linux 2.6.3-7; i686; en_US)KHTML/3.4.89 (like Gecko)
Mozilla/5.0 (compatible; NLNZ_IAHarvester2013 +http://natlib.govt.nz/about-us/current-initiatives/web-harvest-2012)
Mozilla/5.0 (compatible; NLNZ_IAHarvester2013 +http://natlib.govt.nz/about-us/current-initiatives/web-harvest-2013)
Mozilla/5.0 (compatible; NLNZ_IAHarvester2014 +http://natlib.govt.nz/about-us/current-initiatives/web-harvest-2014)
Mozilla/5.0 (compatible; NLNZ_IAHarvester2014 +https://natlib.govt.nz/publishers-and-authors/web-harvesting/domain-harvest)
Mozilla/5.0 (compatible; NLNZ_IAHarvester2016/3.3.0 +https://natlib.govt.nz/publishers-and-authors/web-harvesting/domain-harvest)
Mozilla/5.0 (compatible; NTENTbot; +http://www.ntent.com/ntentbot)
Mozilla/5.0 (compatible; NaverJapan/1.0; +http://corp.naver.jp/)
Mozilla/5.0 (compatible; Neevabot/1.0;  https://neeva.com/neevabot)
Mozilla/5.0 (compatible; NerdByNature.Bot; http://www.nerdbynature.net/bot)
Mozilla/5.0 (compatible; NetcraftSurveyAgent/1.0; +info@netcraft.com)
Mozilla/5.0 (compatible; Netseer crawler/2.0; +http://www.netseer.com/crawler.html; crawler@netseer.com)
Mozilla/5.0 (compatible; Nett.io bot/1.0; +http://nett.io/)
Mozilla/5.0 (compatible; NewShareCounts.com/1.0; +http://newsharecounts.com/crawler)
Mozilla/5.0 (compatible; Nigma.ru/3.0; crawler@nigma.ru)
Mozilla/5.0 (compatible; Nimbostratus-Bot/v1.3.2; http://cloudsystemnetworks.com)
Mozilla/5.0 (compatible; Nmap Scripting Engine; http://nmap.org/book/nse.html)
Mozilla/5.0 (compatible; Nmap Scripting Engine; https://nmap.org/book/nse.html)
Mozilla/5.0 (compatible; OdklBot/1.0 like Linux; klass@odnoklassniki.ru)
Mozilla/5.0 (compatible; Online Domain Tools - HTTP Headers Online/1.0; +http://http-headers.online-domain-tools.com)
Mozilla/5.0 (compatible; Online Domain Tools - Online Sitemap Generator/1.1; +http://sitemap-generator.online-domain-tools.com)
Mozilla/5.0 (compatible; Online Domain Tools - Online Website Link Checker/1.1; +http://website-link-checker.online-domain-tools.com)
Mozilla/5.0 (compatible; Online Domain Tools - Online Website Link Checker/1.2; +http://website-link-checker.online-domain-tools.com)
Mozilla/5.0 (compatible; Online Domain Tools - SEO Checker/1.0; +http://online-domain-tools.com)
Mozilla/5.0 (compatible; Online Domain Tools - Server Monitor/1.0; +http://server-monitoring.online-domain-tools.com)
Mozilla/5.0 (compatible; OpenCrawler/0.1.6.6; http://code.google.com/p/opencrawler/),gzip(gfe),gzip(gfe)
Mozilla/5.0 (compatible; OpenHoseBot/2.1; +http://www.openhose.org/bot.html)
Mozilla/5.0 (compatible; OpenX Spider; http://www.openx.org)
Mozilla/5.0 (compatible; OpenfosBot/2.1; http://www.openfos.com)
Mozilla/5.0 (compatible; OpenfosBot/2.4; +http://www.openfos.com)
Mozilla/5.0 (compatible; OpenindexDeepSpider/Nutch-1.5-dev; +http://openindex.io/spider.html; systemsATopenindexDOTio)
Mozilla/5.0 (compatible; OpenindexDeepSpider/Nutch-1.5-dev; +http://www.openindex.io/en/webmasters/spider.html)
Mozilla/5.0 (compatible; OpenindexDeepSpider/Nutch-1.5-dev; +http://www.openindex.io/en/webmasters/spider.html; systemsATopenindexDOTio)
Mozilla/5.0 (compatible; OpenindexShallowSpider/Nutch-1.5-dev; +http://www.openindex.io/en/webmasters/spider.html)
Mozilla/5.0 (compatible; OpenindexShallowSpider/Nutch-1.5-dev; +http://www.openindex.io/en/webmasters/spider.html; systemsATopenindexDOTio)
Mozilla/5.0 (compatible; OpenindexSpider/Nutch-1.5-dev; +http://www.openindex.io/en/webmasters/spider.html)
Mozilla/5.0 (compatible; OpenindexSpider; +http://www.openindex.io/en/webmasters/spider.html)
Mozilla/5.0 (compatible; OptimizationCrawler/0.2; +http://www.domainoptima.com/bot.html)
Mozilla/5.0 (compatible; OptimizationCrawler/0.2; +http://www.domainoptima.com/robot)
Mozilla/5.0 (compatible; Optimizer)
Mozilla/5.0 (compatible; OrangeBot-Collector/2.0; support.orangebot@orange.com)
Mozilla/5.0 (compatible; OrangeBot/2.0; support.orangebot@orange.com
Mozilla/5.0 (compatible; OrangeBot/2.0; support.orangebot@orange.com)
Mozilla/5.0 (compatible; OrangeBot/2.0; support.voilabot@orange.com)
Mozilla/5.0 (compatible; Orangebot/2.0; support.orangebot@orange.com)
Mozilla/5.0 (compatible; OsO; http://oso.octopodus.com/abot.html)
Mozilla/5.0 (compatible; PAD-bot/9.0; +http://www.descargarprogramagratis.com/)
Mozilla/5.0 (compatible; PR-CY.RU; + https://a.pr-cy.ru)
Mozilla/5.0 (compatible; PRTG Network Monitor (www.paessler.com); Windows)
Mozilla/5.0 (compatible; Pagespeed/1.1 Fetcher; +http://www.pagespeed.de)
Mozilla/5.0 (compatible; Pandeo Bot; +http://pandeo.de/bot.php)
Mozilla/5.0 (compatible; PaperLiBot/2.1; http://support.paper.li/entries/20023257-what-is-paper-li)
Mozilla/5.0 (compatible; PaperLiBot/2.1; https://support.paper.li/entries/20023257-what-is-paper-li)
Mozilla/5.0 (compatible; ParsijooBot; +http://www.parsijoo.ir/; crawler@parsijoo.ir)
Mozilla/5.0 (compatible; PathDefender/1.0; +http://www.pathdefender.com/help/crawler)
Mozilla/5.0 (compatible; PaxleFramework/0.1.1; http://www.paxle.net/en/bot)
Mozilla/5.0 (compatible; Peepowbot/1.0; +http://www.peepow.com/bot.php)
Mozilla/5.0 (compatible; Peew/1.0; http://www.peew.de/crawler/)
Mozilla/5.0 (compatible; Pi-Monster; https://pricepi.com/)
Mozilla/5.0 (compatible; PingAdmin.Ru/1.1; +http://pingadmin.ru/)
Mozilla/5.0 (compatible; PingAdmin.Ru/1.2; +http://pingadmin.ru/free_test/)
Mozilla/5.0 (compatible; Pinterestbot/1.0; +http://www.pinterest.com/bot.html)
Mozilla/5.0 (compatible; Pinterestbot/1.0; http://www.pinterest.com/bot.html)
Mozilla/5.0 (compatible; PiplBot; http://www.pipl.com/bot/)
Mozilla/5.0 (compatible; Plukkie/1.1; http://www.botje.com/plukkie.htm)
Mozilla/5.0 (compatible; Plukkie/1.2; http://www.botje.com/plukkie.htm)
Mozilla/5.0 (compatible; Plukkie/1.3; http://www.botje.com/plukkie.htm)
Mozilla/5.0 (compatible; Plukkie/1.4; http://www.botje.com/plukkie.htm)
Mozilla/5.0 (compatible; Plukkie/1.5; http://www.botje.com/plukkie.htm)
Mozilla/5.0 (compatible; Plukkie/1.6; http://www.botje.com/plukkie.htm)
Mozilla/5.0 (compatible; PornSpider/1.0; +http://www.pornspider.net)
Mozilla/5.0 (compatible; Primalbot; +https://www.primal.com;)
Mozilla/5.0 (compatible; PrivacyAwareBot/1.1; +http://www.privacyaware.org)
Mozilla/5.0 (compatible; Prlog/1.0; +http://prlog.ru/)
Mozilla/5.0 (compatible; Pro Sitemaps Generator; https://pro-sitemaps.com) Gecko Pro-Sitemaps/1.0
Mozilla/5.0 (compatible; ProCogBot/1.0; +http://www.procog.com/spider.html)
Mozilla/5.0 (compatible; ProCogSEOBot/1.0; +http://www.procog.com/ )
Mozilla/5.0 (compatible; ProductoDownloadUrlBot/1.0; +http://www.producto.de/)
Mozilla/5.0 (compatible; ProspectB2B/2.0; http://prospectb2b.com)
Mozilla/5.0 (compatible; Qseero; +http://www.q0.com)
Mozilla/5.0 (compatible; Qualidator.com Bot 1.0;)
Mozilla/5.0 (compatible; Qualidator.com SiteAnalyzer 1.0;)
Mozilla/5.0 (compatible; Qwantify/2.0; +https://www.qwant.com/)
Mozilla/5.0 (compatible; Qwantify/2.0n; +https://www.qwant.com/)
Mozilla/5.0 (compatible; Qwantify/2.0n; +https://www.qwant.com/)/*
Mozilla/5.0 (compatible; Qwantify/2.1dw; +https://www.qwant.com/)/*
Mozilla/5.0 (compatible; Qwantify/2.1n; +https://www.qwant.com/)/*
Mozilla/5.0 (compatible; Qwantify/2.1w; +https://www.qwant.com/)/*
Mozilla/5.0 (compatible; Qwantify/2.2w; +https://www.qwant.com/)/*
Mozilla/5.0 (compatible; Qwantify/2.3w; +https://www.qwant.com/)/2.3w
Mozilla/5.0 (compatible; Qwantify/2.4w; +https://www.qwant.com/)/2.4w
Mozilla/5.0 (compatible; Qwantify/Bleriot/1.1; +https://help.qwant.com/bot)
Mozilla/5.0 (compatible; Qwantify/Bleriot/1.2.1; +https://help.qwant.com/bot)
Mozilla/5.0 (compatible; RTGI; http://linkfluence.net/)
Mozilla/5.0 (compatible; Rakutenbot/1.0; +http://dynamic.rakuten.co.jp/bot.html)
Mozilla/5.0 (compatible; RankActiveLinkBot; +https://rankactive.com/resources/rankactive-linkbot)
Mozilla/5.0 (compatible; RankSonicSiteAuditor/1.0; +https://ranksonic.com/ranksonic_sab.html)
Mozilla/5.0 (compatible; RankvalBot/1.0; +https://rankval.com)
Mozilla/5.0 (compatible; RavenCrawler/2.0; +https://raventools.com/seo-website-auditor/)
Mozilla/5.0 (compatible; RegionStuttgartBot/1.0; +http://it.region-stuttgart.de/competenzatlas/unternehmen-suchen/)
Mozilla/5.0 (compatible; RelateIQ Crawler/0.4; +www.relateiq.com; info@relateiq.com)
Mozilla/5.0 (compatible; RidderBot/1.0; bot@ridder.co)
Mozilla/5.0 (compatible; RidderBot/1.0; bot@ridder.co) (iPhone; CPU iPhone OS 8_4_1 like Mac OS X) Mobile/12H321
Mozilla/5.0 (compatible; Rivva; http://rivva.de)
Mozilla/5.0 (compatible; RukiCrawler/1.0; +http://ruki.rezko.net)
Mozilla/5.0 (compatible; RyowlEngine/1.0; +https://ryowl.org)
Mozilla/5.0 (compatible; SEODat/0.1 http://crawler.seodat.com)
Mozilla/5.0 (compatible; SEOdiver/1.0; +http://www.seodiver.com/bot)
Mozilla/5.0 (compatible; SEOkicks-Robot +http://www.seokicks.de/robot.html)
Mozilla/5.0 (compatible; SEOkicks-Robot; +http://www.seokicks.de/robot.html)
Mozilla/5.0 (compatible; SEOkicks; +https://www.seokicks.de/robot.html)
Mozilla/5.0 (compatible; SEOlyticsCrawler/3.0; +http://crawler.seolytics.net/)
Mozilla/5.0 (compatible; SISTRIX Crawler; http://crawler.sistrix.net/)
Mozilla/5.0 (compatible; SISTRIX Optimizer)
Mozilla/5.0 (compatible; SISTRIX Optimizer; Uptime; +https://www.sistrix.com/faq/uptime)
Mozilla/5.0 (compatible; SMTBot/1.0; +http://www.similartech.com/smtbot)
Mozilla/5.0 (compatible; SMTBot/1.0; http://www.similartech.com/smtbot)
Mozilla/5.0 (compatible; SOLOFIELD/1.0 +http://solofield.net/bot.html)
Mozilla/5.0 (compatible; SRCCN!Spider/1.1; +http://site.srccn.com/spider.html)
Mozilla/5.0 (compatible; SSLBot/1.0;  http://www.sslstats.com/sslbot)
Mozilla/5.0 (compatible; STINGbot/1.0; +http://136.186.231.16)
Mozilla/5.0 (compatible; SWEBot/1.0; +http://swebot-crawler.net)
Mozilla/5.0 (compatible; SWEBot/1.0; +http://swebot.net)
Mozilla/5.0 (compatible; SabsimBot/3.0; +https://sabsim.com)
Mozilla/5.0 (compatible; Scarlett/ 1.0; +http://www.ellerdale.com/crawler.html)
Mozilla/5.0 (compatible; Scopia Crawler 1.1; +http://www.scopia.co)
Mozilla/5.0 (compatible; Scopia Crawler 1.2; +http://www.scopia.co)
Mozilla/5.0 (compatible; Scopia Crawler 2.0; +http://www.scopia.co)
Mozilla/5.0 (compatible; ScoutJet; +http://www.scoutjet.com/)
Mozilla/5.0 (compatible; ScoutJet; http://www.scoutjet.com/)
Mozilla/5.0 (compatible; Scrubby/2.2; http://www.scrubtheweb.com/)
Mozilla/5.0 (compatible; Scrubby/3.1; +http://www.scrubtheweb.com/help/technology.html)
Mozilla/5.0 (compatible; Scrubby/3.2; +http://seotools.scrubtheweb.com/webpage-analyzer.html)
Mozilla/5.0 (compatible; Scrutiny/8; +https://peacockmedia.software/mac/scrutiny/
Mozilla/5.0 (compatible; Search17Bot/1.1; http://www.search17.com/bot.php)
Mozilla/5.0 (compatible; SearchmetricsBot; http://www.searchmetrics.com/en/searchmetrics-bot/)
Mozilla/5.0 (compatible; SecretSerachEngineLabs.com-SBSearch/0.9; http://www.secretsearchenginelabs.com/secret-web-crawler.php)
Mozilla/5.0 (compatible; SecurityResearch.bot; +http://besome1.info/securityresearchbot.html)
Mozilla/5.0 (compatible; Seekport Crawler; http://seekport.com/)
Mozilla/5.0 (compatible; Semager/1.4; http://www.semager.de/blog/semager-bots/)
Mozilla/5.0 (compatible; Semager/1.4c; +http://www.semager.de/blog/semager-bots/)
Mozilla/5.0 (compatible; SemanticBot/1.1)
Mozilla/5.0 (compatible; Semanticbot/1.0; +http://sempi.tech/bot.html)
Mozilla/5.0 (compatible; Semiocast HTTP client; http://semiocast.com/)
Mozilla/5.0 (compatible; SemrushBot-BA; +http://www.semrush.com/bot.html)
Mozilla/5.0 (compatible; SemrushBot-SA/0.97; +http://www.semrush.com/bot.html)
Mozilla/5.0 (compatible; SemrushBot-SI/0.97; +http://www.semrush.com/bot.html)
Mozilla/5.0 (compatible; SemrushBot/0.95; +http://www.semrush.com/bot.html)
Mozilla/5.0 (compatible; SemrushBot/0.96.2; +http://www.semrush.com/bot.html)
Mozilla/5.0 (compatible; SemrushBot/0.96.3; +http://www.semrush.com/bot.html)
Mozilla/5.0 (compatible; SemrushBot/0.96.4; +http://www.semrush.com/bot.html)
Mozilla/5.0 (compatible; SemrushBot/0.97; +http://www.semrush.com/bot.html)
Mozilla/5.0 (compatible; SemrushBot/0.97sa; +http://www.semrush.com/bot.html)
Mozilla/5.0 (compatible; SemrushBot/0.97~bl; +http://www.semrush.com/bot.html)
Mozilla/5.0 (compatible; SemrushBot/0.98~bl; +http://www.semrush.com/bot.html)
Mozilla/5.0 (compatible; SemrushBot/0.99~bl; +http://www.semrush.com/bot.html)
Mozilla/5.0 (compatible; SemrushBot/1.1~bl; +http://www.semrush.com/bot.html)
Mozilla/5.0 (compatible; SemrushBot/1.2~bl; +http://www.semrush.com/bot.html)
Mozilla/5.0 (compatible; SemrushBot/1~bl; +http://www.semrush.com/bot.html)
Mozilla/5.0 (compatible; SemrushBot/3~bl; +http://www.semrush.com/bot.html)
Mozilla/5.0 (compatible; SemrushBot/6~bl; +http://www.semrush.com/bot.html)
Mozilla/5.0 (compatible; SetLinks bot 1.2)
Mozilla/5.0 (compatible; Seznam screenshot-generator 2.0; +http://fulltext.sblog.cz/screenshot/)
Mozilla/5.0 (compatible; Seznam screenshot-generator 2.1; +http://fulltext.sblog.cz/screenshot/)
Mozilla/5.0 (compatible; SeznamBot/3.1-test1; +http://fulltext.sblog.cz/)
Mozilla/5.0 (compatible; SeznamBot/3.2-test1-1; +http://napoveda.seznam.cz/en/seznambot-intro/)
Mozilla/5.0 (compatible; SeznamBot/3.2-test1; +http://fulltext.sblog.cz/)
Mozilla/5.0 (compatible; SeznamBot/3.2-test1; +http://napoveda.seznam.cz/en/seznambot-intro/)
Mozilla/5.0 (compatible; SeznamBot/3.2-test2; +http://fulltext.sblog.cz/)
Mozilla/5.0 (compatible; SeznamBot/3.2-test2; +http://napoveda.seznam.cz/en/seznambot-intro/)
Mozilla/5.0 (compatible; SeznamBot/3.2-test3; +http://fulltext.sblog.cz/)
Mozilla/5.0 (compatible; SeznamBot/3.2-test4; +http://fulltext.sblog.cz/)
Mozilla/5.0 (compatible; SeznamBot/3.2-test4; +http://napoveda.seznam.cz/en/seznambot-intro/)
Mozilla/5.0 (compatible; SeznamBot/3.2; +http://fulltext.sblog.cz/)
Mozilla/5.0 (compatible; SeznamBot/3.2; +http://napoveda.seznam.cz/en/seznambot-intro/)
Mozilla/5.0 (compatible; Shareaholicbot/1.0; +http://www.shareaholic.com/bot)
Mozilla/5.0 (compatible; Shareaholicbot/2.0; +http://www.shareaholic.com/bot)
Mozilla/5.0 (compatible; SimpleScraper)
Mozilla/5.0 (compatible; SiteAnalyzerBot/4.0; +https://www.site-analyzer.com/)
Mozilla/5.0 (compatible; SiteBot/0.1; +http://www.sitebot.org/robot/)
Mozilla/5.0 (compatible; SiteCondor; http://www.sitecondor.com)
Mozilla/5.0 (compatible; SiteExplorer/1.0b; +http://siteexplorer.info/)
Mozilla/5.0 (compatible; SiteExplorer/1.1b; +http://siteexplorer.info/Backlink-Checker-Spider/)
Mozilla/5.0 (compatible; SiteIndexed/1.2; +http://www.siteindexed.com)
Mozilla/5.0 (compatible; Siteliner/1.0; +http://www.siteliner.com/bot)
Mozilla/5.0 (compatible; Siteluxbot/1.0; +https://sitelux.co.uk/)
Mozilla/5.0 (compatible; Sitemap Generator/1.3; +http://www.check-domains.com/sitemap/index.php)
Mozilla/5.0 (compatible; Sitemap Generator/1.3; http://www.check-domains.com/sitemap/index.php) Gecko Check-domains)
Mozilla/5.0 (compatible; Snacktory; +https://flynxapp.com)
Mozilla/5.0 (compatible; Snacktory; +https://github.com/karussell/snacktory)
Mozilla/5.0 (compatible; SoftListBot/2.2; +http://www.softlist.us/)
Mozilla/5.0 (compatible; Sonic/1.0; http://www.yama.info.waseda.ac.jp/~crawler/info.html)
Mozilla/5.0 (compatible; Sophora Linkchecker; http://www.subshell.com)
Mozilla/5.0 (compatible; Sosospider/2.0; +http://help.soso.com/webspider.htm)
Mozilla/5.0 (compatible; Speedy Spider; http://www.entireweb.com/about/search_tech/speedy_spider/)
Mozilla/5.0 (compatible; SpiderLing (a SPIDER for LINGustic research); +http://nlp.fi.muni.cz/projects/biwec/)
Mozilla/5.0 (compatible; SpiderLing (a SPIDER for LINGustic research); http://nlp.fi.muni.cz/projects/biwec/)
Mozilla/5.0 (compatible; Spiderlytics/1.0; +spider@spiderlytics.com)
Mozilla/5.0 (compatible; SputnikBot/2.2)
Mozilla/5.0 (compatible; SputnikBot/2.3; +http://corp.sputnik.ru/webmaster)
Mozilla/5.0 (compatible; SputnikFaviconBot/1.2; +http://corp.sputnik.ru/webmaster)
Mozilla/5.0 (compatible; SputnikImageBot/2.2)
Mozilla/5.0 (compatible; SputnikImageBot/2.3; +http://corp.sputnik.ru/webmaster)
Mozilla/5.0 (compatible; Squzer/0.0.1; +http://www.declum.com/)
Mozilla/5.0 (compatible; Steeler/3.5; http://www.tkl.iis.u-tokyo.ac.jp/~crawler/)
Mozilla/5.0 (compatible; StoneSunSpider/1.1)
Mozilla/5.0 (compatible; StorygizeBot; http://www.storygize.com)
Mozilla/5.0 (compatible; StreamScraper/1.0; +http://code.google.com/p/streamscraper/)
Mozilla/5.0 (compatible; SurdotlyBot/1.0; +http://sur.ly/bot.html)
Mozilla/5.0 (compatible; SurdotlyBot/1.0; +http://sur.ly/bot.html; Linux; Android 4; iPhone; CPU iPhone OS 6_0_1 like Mac OS X)
Mozilla/5.0 (compatible; SurdotlyBot/1.0; +http://sur.ly/bot.html; Linux; Android 4; iPhone; CPU iPhone OS 6_0_1 like Mac OS X) AppleWebKit/536.26 (KHTML, like Gecko) Version/6.0 Mobile/10A523 Safari/8536.25
Mozilla/5.0 (compatible; SwiteScraper)
Mozilla/5.0 (compatible; Sysomos/1.0; +http://www.sysomos.com/; Sysomos)
Mozilla/5.0 (compatible; TWMBot/0.1; +http://thewebminer.com)
Mozilla/5.0 (compatible; Taboolabot/3.7; +http://www.taboola.com)
Mozilla/5.0 (compatible; Tagoobot/3.0; +http://www.tagoo.ru)
Mozilla/5.0 (compatible; TeeRaidBot; +https://teeraid.com/bot/)
Mozilla/5.0 (compatible; TestCrawler; +http://wiki.github.com/bixo/bixo/bixocrawler; bixo-dev@yahoogroups.com)
Mozilla/5.0 (compatible; TestURI; +http://testuri.org/)
Mozilla/5.0 (compatible; Tetrahedron/1.0; +https://compute.info/tet)
Mozilla/5.0 (compatible; The Drop Reaper; +http://tag1consulting.com/dropreaper.html)
Mozilla/5.0 (compatible; ThinkChaos/0.3.0; +In_the_test_phase,_if_the_ThinkChaos_brings_you_trouble,_please_add_disallow_to_the_robots.txt._Thank_you.)
Mozilla/5.0 (compatible; Thumbshots.ru; +http://thumbshots.ru/bot) Firefox/3
Mozilla/5.0 (compatible; TinEye-bot/1.31; +http://www.tineye.com/crawler.html)
Mozilla/5.0 (compatible; Topicbot/12.0rc-2; +http://topicbot.awardspace.us/)
Mozilla/5.0 (compatible; ToutiaoSpider/1.0; http://web.toutiao.com/media_cooperation/;)
Mozilla/5.0 (compatible; TrendsmapResolver/0.1)
Mozilla/5.0 (compatible; TridentSpider/3.1)
Mozilla/5.0 (compatible; TweetedTimes Bot/1.0; +http://tweetedtimes.com)
Mozilla/5.0 (compatible; TweetmemeBot/2.11; +http://tweetmeme.com/)
Mozilla/5.0 (compatible; TweetmemeBot/3.0; +http://tweetmeme.com/)
Mozilla/5.0 (compatible; Tweezler/2.0; http://tweezler.com)
Mozilla/5.0 (compatible; Twingly Recon; twingly.com)
Mozilla/5.0 (compatible; TwitterCrawler)
Mozilla/5.0 (compatible; U; AnyEvent-HTTP/2.22; +http://software.schmorp.de/pkg/AnyEvent)
Mozilla/5.0 (compatible; U; AnyEvent-HTTP/2.23; +http://software.schmorp.de/pkg/AnyEvent)
Mozilla/5.0 (compatible; U; AnyEvent-HTTP/2.24; +http://software.schmorp.de/pkg/AnyEvent)
Mozilla/5.0 (compatible; U; Pcore-HTTP-UA/v0.12.4
Mozilla/5.0 (compatible; UASlinkChecker/1.0; +http://user-agent-string.info/UASlinkChecker)
Mozilla/5.0 (compatible; UASlinkChecker/2.0; +http://udger.com/support/UASlinkChecker)
Mozilla/5.0 (compatible; UASlinkChecker/2.1; +https://udger.com/support/UASlinkChecker)
Mozilla/5.0 (compatible; URLAppendBot/1.0; +http://www.profound.net/urlappendbot.html)
Mozilla/5.0 (compatible; UnisterBot; crawler@unister.de)
Mozilla/5.0 (compatible; UnisterBot; http://www.bluekiwi.de/misc/imprint)
Mozilla/5.0 (compatible; UptimeRobot/1.0; http://www.uptimerobot.com/)
Mozilla/5.0 (compatible; UptimeRobot/2.0; http://www.uptimerobot.com/)
Mozilla/5.0 (compatible; Uptimebot/0.1.38; +http://www.uptime.com/uptimebot)
Mozilla/5.0 (compatible; Uptimebot/0.1.43; +http://www.uptime.com/uptimebot)
Mozilla/5.0 (compatible; Uptimebot/0.1.73; +http://www.uptime.com/uptimebot)
Mozilla/5.0 (compatible; Uptimebot/0.1.89; +http://www.uptime.com/uptimebot)
Mozilla/5.0 (compatible; Uptimebot/0.1.97; +http://www.uptime.com/uptimebot)
Mozilla/5.0 (compatible; Uptimebot/0.2.14; +http://www.uptime.com/uptimebot)
Mozilla/5.0 (compatible; Uptimebot/0.2.18; +http://www.uptime.com/uptimebot)
Mozilla/5.0 (compatible; Uptimebot/0.2.19; +http://www.uptime.com/uptimebot)
Mozilla/5.0 (compatible; Uptimebot/0.2.22; +http://www.uptime.com/uptimebot)
Mozilla/5.0 (compatible; Uptimebot/0.2.29; +http://www.uptime.com/uptimebot)
Mozilla/5.0 (compatible; Uptimebot/0.2.40; +http://www.uptime.com/uptimebot)
Mozilla/5.0 (compatible; Uptimebot/0.2.41; +http://www.uptime.com/uptimebot)
Mozilla/5.0 (compatible; Uptimebot/0.2.42; +http://www.uptime.com/uptimebot)
Mozilla/5.0 (compatible; Uptimebot/0.2.43; +http://www.uptime.com/uptimebot)
Mozilla/5.0 (compatible; Uptimebot/1.0; +http://www.uptime.com/uptimebot)
Mozilla/5.0 (compatible; Urlfilebot/2.2; +http://urlfile.com/bot.html)
Mozilla/5.0 (compatible; VKRobot/1.0)
Mozilla/5.0 (compatible; VSMCrawler; http://www.visualsitemapper.com/crawler/)
Mozilla/5.0 (compatible; Vagabondo/2.1; webcrawler at wise-guys dot nl; http://webagent.wise-guys.nl/)
Mozilla/5.0 (compatible; VelenPublicWebCrawler/1.0; +https://velen.io)
Mozilla/5.0 (compatible; Veoozbot/1.0; +http://www.veooz.com/veoozbot.html)
Mozilla/5.0 (compatible; VideoSurf_bot +http://www.videosurf.com/bot.html)
Mozilla/5.0 (compatible; Vigil/1.0; +http://vigil-app.com/bot.html)
Mozilla/5.0 (compatible; VodpodCrawler/1.0; +http://vodpod.com/site/help)
Mozilla/5.0 (compatible; VoluumDSP-content-bot/2.0; +dsp-dev@codewise.com)
Mozilla/5.0 (compatible; WASALive-Bot ; http://blog.wasalive.com/wasalive-bots/)
Mozilla/5.0 (compatible; WBSearchBot/1.1; +http://www.warebay.com/bot.html)
Mozilla/5.0 (compatible; WangIDSpider/1.0; +http://www.wangid.com/spider.html)
Mozilla/5.0 (compatible; Wappalyzer; +https://github.com/AliasIO/Wappalyzer)
Mozilla/5.0 (compatible; Wappalyzer; https://www.wappalyzer.com)
Mozilla/5.0 (compatible; WbSrch/1.0; +https://wbsrch.com)
Mozilla/5.0 (compatible; WbSrch/1.1 +http://wbsrch.com
Mozilla/5.0 (compatible; WbSrch/1.1 +http://wbsrch.com)
Mozilla/5.0 (compatible; WeViKaBot/1.0; +http://www.wevika.de/)
Mozilla/5.0 (compatible; Web-Monitoring/1.0; +http://monoid.nic.ru/)
Mozilla/5.0 (compatible; WebCookies/1.0; +http://webcookies.org/faq/#agent)
Mozilla/5.0 (compatible; WebCookies/1.0; +https://webcookies.org/faq/#agent)
Mozilla/5.0 (compatible; WebCorp/5.0; +http://www.webcorp.org.uk)
Mozilla/5.0 (compatible; WebCrunch/1.2; +http://webcrunch.net/crawler)
Mozilla/5.0 (compatible; WebDataStats/1.0 ; +https://webdatastats.com/policy.html)
Mozilla/5.0 (compatible; WebMasterAid/1.0; +http://wmaid.com/explore/)
Mozilla/5.0 (compatible; WebNL; +http://www.web.nl/webmasters/spider.html; helpdesk@web.nl)
Mozilla/5.0 (compatible; WebThumbnail/2.2; Website Thumbnail Generat
Mozilla/5.0 (compatible; WebThumbnail/2.2; Website Thumbnail Generator; +http://webthumbnail.org)
Mozilla/5.0 (compatible; WebThumbnail/3.x; Website Thumbnail Generator; +http://webthumbnail.org)
Mozilla/5.0 (compatible; Webauskunft.org; http://www.webauskunft.org)
Mozilla/5.0 (compatible; Webbot/0.1; http://www.webbot.ru/bot.html)
Mozilla/5.0 (compatible; Webmaster tools +http://sitexy.com/)
Mozilla/5.0 (compatible; WebmasterCoffee/0.7; +http://webmastercoffee.com/about)
Mozilla/5.0 (compatible; Website Analyzer/1.1; +http://www.check-domains.com/website-analysis/website-analyzer.php)
Mozilla/5.0 (compatible; Whoiswebsitebot/0.1; +http://www.whoiswebsite.net)
Mozilla/5.0 (compatible; Whynder Magnet; http://whynder.com/magnet)
Mozilla/5.0 (compatible; Windows NT 6.1?; ZumBot/1.0; http://help.zum.com/inquiry)
Mozilla/5.0 (compatible; WoTBoT; +https://www.wslta.com/WoTBoT.html)
Mozilla/5.0 (compatible; WormlyBot; +http://wormly.com)
Mozilla/5.0 (compatible; X11; Linux x86_64; Google-StructuredDataTestingTool; +http://www.google.com/webmasters/tools/richsnippets)
Mozilla/5.0 (compatible; XML Sitemaps Generator; http://www.xml-sitemaps.com) Gecko XML-Sitemaps/1.0
Mozilla/5.0 (compatible; XML Sitemaps Generator; https://www.xml-sitemaps.com) Gecko XML-Sitemaps/1.0
Mozilla/5.0 (compatible; XmarksFetch/1.0; +http://www.xmarks.com/about/crawler; info@xmarks.com)
Mozilla/5.0 (compatible; XoviBot/2.0; +http://www.xovibot.net/)
Mozilla/5.0 (compatible; Y!J SearchMonkey/1.0 (Y!J-AGENT; http://help.yahoo.co.jp/help/jp/search/indexing/indexing-15.html))
Mozilla/5.0 (compatible; YRSpider; +http://www.yunrang.com/yrspider.html)
Mozilla/5.0 (compatible; YRSpider; +http://www.yunyun.com/SiteInfo.php?r=about)
Mozilla/5.0 (compatible; YYSpider; +http://www.yunyun.com/spider.html)
Mozilla/5.0 (compatible; YaDirectFetcher/1.0; +http://yandex.com/bots)
Mozilla/5.0 (compatible; YaDirectFetcher/1.0; Dyatel; +http://yandex.com/bots)
Mozilla/5.0 (compatible; YaK/1.0; http://linkfluence.com/; bot@linkfluence.com)
Mozilla/5.0 (compatible; Yahoo Ad monitoring; https://help.yahoo.com/kb/yahoo-ad-monitoring-SLN24857.html)
Mozilla/5.0 (compatible; Yahoo Link Preview; https://help.yahoo.com/kb/mail/yahoo-link-preview-SLN23615.html)
Mozilla/5.0 (compatible; Yahoo! Slurp China; http://misc.yahoo.com.cn/help.html)
Mozilla/5.0 (compatible; Yahoo! Slurp/3.0; http://help.yahoo.com/help/us/ysearch/slurp)
Mozilla/5.0 (compatible; Yahoo! Slurp/3.0; http://help.yahoo.com/help/us/ysearch/slurp) NOT Firefox/3.5
Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)
Mozilla/5.0 (compatible; YandexAccessibilityBot/3.0; +http://yandex.com/bots
Mozilla/5.0 (compatible; YandexAccessibilityBot/3.0; +http://yandex.com/bots)
Mozilla/5.0 (compatible; YandexAdNet/1.0; +http://yandex.com/bots)
Mozilla/5.0 (compatible; YandexAntivirus/2.0; +http://yandex.com/bots)
Mozilla/5.0 (compatible; YandexBlogs/0.99; robot; +http://yandex.com/bots)
Mozilla/5.0 (compatible; YandexBlogs/0.99; robot; B; +http://yandex.com/bots)
Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)
Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots),gzip(gfe) AppEngine-Google; (+http://code.google.com/appengine; appid: twitter-mirror),gzip(gfe),gzip(gfe),gzip(gfe)
Mozilla/5.0 (compatible; YandexBot/3.0; MirrorDetector; +http://yandex.com/bots)
Mozilla/5.0 (compatible; YandexCalendar/1.0; +http://yandex.com/bots)
Mozilla/5.0 (compatible; YandexCatalog/3.0; +http://yandex.com/bots)
Mozilla/5.0 (compatible; YandexDirect/3.0; +http://yandex.com/bots)
Mozilla/5.0 (compatible; YandexDirectDyn/1.0; +http://yandex.com/bots
Mozilla/5.0 (compatible; YandexFavicons/1.0; +http://yandex.com/bots)
Mozilla/5.0 (compatible; YandexForDomain/1.0; +http://yandex.com/bots)
Mozilla/5.0 (compatible; YandexImageResizer/2.0; +http://yandex.com/bots)
Mozilla/5.0 (compatible; YandexImages/3.0; +http://yandex.com/bots)
Mozilla/5.0 (compatible; YandexMarket/1.0; +http://yandex.com/bots)
Mozilla/5.0 (compatible; YandexMarket/2.0; +http://yandex.com/bots)
Mozilla/5.0 (compatible; YandexMedia/3.0; +http://yandex.com/bots)
Mozilla/5.0 (compatible; YandexMetrika/2.0; +http://yandex.com/bots DEV)
Mozilla/5.0 (compatible; YandexMetrika/2.0; +http://yandex.com/bots mtmon01e.yandex.ru)
Mozilla/5.0 (compatible; YandexMetrika/2.0; +http://yandex.com/bots mtmon01g.yandex.ru)
Mozilla/5.0 (compatible; YandexMetrika/2.0; +http://yandex.com/bots mtmon01i.yandex.ru)
Mozilla/5.0 (compatible; YandexMetrika/2.0; +http://yandex.com/bots mtmon01it.mtrs.yandex.ru)
Mozilla/5.0 (compatible; YandexMetrika/2.0; +http://yandex.com/bots mtweb01t.yandex.ru)
Mozilla/5.0 (compatible; YandexMetrika/2.0; +http://yandex.com/bots yabs01)
Mozilla/5.0 (compatible; YandexMetrika/2.0; +http://yandex.com/bots)
Mozilla/5.0 (compatible; YandexMetrika/3.0; +http://yandex.com/bots)
Mozilla/5.0 (compatible; YandexMetrika/4.0; +http://yandex.com/bots)
Mozilla/5.0 (compatible; YandexMobileScreenShotBot/1.0; +http://yandex.com/bots)
Mozilla/5.0 (compatible; YandexNews/3.0; +http://yandex.com/bots)
Mozilla/5.0 (compatible; YandexNews/4.0; +http://yandex.com/bots)
Mozilla/5.0 (compatible; YandexNewslinks; +http://yandex.com/bots)
Mozilla/5.0 (compatible; YandexOntoDB/1.0; +http://yandex.com/bots)
Mozilla/5.0 (compatible; YandexOntoDBAPI/1.0; +http://yandex.com/bots)
Mozilla/5.0 (compatible; YandexPagechecker/1.0; +http://yandex.com/bots)
Mozilla/5.0 (compatible; YandexPagechecker/2.0; +http://yandex.com/bots)
Mozilla/5.0 (compatible; YandexPartner/3.0; +http://yandex.com/bots)
Mozilla/5.0 (compatible; YandexRCA/1.0; +http://yandex.com/bots)
Mozilla/5.0 (compatible; YandexSearchShop/1.0; +http://yandex.com/bots)
Mozilla/5.0 (compatible; YandexSitelinks; Dyatel; +http://yandex.com/bots)
Mozilla/5.0 (compatible; YandexSpravBot/1.0; +http://yandex.com/bots)
Mozilla/5.0 (compatible; YandexTracker/1.0; +http://yandex.com/bots)
Mozilla/5.0 (compatible; YandexTurbo/1.0; +http://yandex.com/bots)
Mozilla/5.0 (compatible; YandexVerticals/1.0; +http://yandex.com/bots)
Mozilla/5.0 (compatible; YandexVerticals/1.0; http://yandex.com/bots)
Mozilla/5.0 (compatible; YandexVertis/3.0; +http://yandex.com/bots)
Mozilla/5.0 (compatible; YandexVideo/3.0; +http://yandex.com/bots)
Mozilla/5.0 (compatible; YandexVideoParser/1.0; +http://yandex.com/bots)
Mozilla/5.0 (compatible; YandexWebmaster/2.0; +http://yandex.com/bots)
Mozilla/5.0 (compatible; YandexZakladki/3.0; +http://yandex.com/bots)
Mozilla/5.0 (compatible; YesupBot/1.0; +http://www.yesup.net/bot.html)
Mozilla/5.0 (compatible; Yeti/1.1; +http://help.naver.com/robots/)
Mozilla/5.0 (compatible; Yeti/1.1; +http://help.naver.com/support/robots.html)
Mozilla/5.0 (compatible; Yeti/1.1; +http://naver.me/bot)
Mozilla/5.0 (compatible; YioopBot; +http://173.13.143.74/bot.php)
Mozilla/5.0 (compatible; YioopBot; +http://www.yioop.com/bot.php)
Mozilla/5.0 (compatible; YodaoBot/1.0; http://www.yodao.com/help/webmaster/spider/; )
Mozilla/5.0 (compatible; YoudaoBot-rts/1.0; http://www.youdao.com/help/webmaster/spider/; )
Mozilla/5.0 (compatible; YoudaoBot/1.0; http://www.youdao.com/help/webmaster/spider/; )
Mozilla/5.0 (compatible; Your-Website-Sucks/2.0; +http://www.Your-Website-Sucks.net/WWW.SOMEWEBSITE.COM)
Mozilla/5.0 (compatible; ZB-1; +http://www.zeerch.com/bot.php)
Mozilla/5.0 (compatible; ZemlyaCrawl/1.0; +http://zemlyaozer.com/bot)
Mozilla/5.0 (compatible; ZumBot/1.0; http://help.zum.com/inquiry)
Mozilla/5.0 (compatible; ZuperlistBot/1.0)
Mozilla/5.0 (compatible; abby/1.0; +http://www.ellerdale.com/crawler.html)
Mozilla/5.0 (compatible; adbeat-publisher-description-fetcher; +crawler@scaleunlimited.com; crawler@scaleunlimited.com)
Mozilla/5.0 (compatible; adbeat_bot; +support@adbeat.com; support@adbeat.com)
Mozilla/5.0 (compatible; adidxbot/2.0;  http://www.bing.com/bingbot.htm)
Mozilla/5.0 (compatible; adidxbot/2.0; +http://www.bing.com/bingbot.htm)
Mozilla/5.0 (compatible; adscanner/)
Mozilla/5.0 (compatible; aiHitBot-BP/1.1; +http://www.aihit.com/)
Mozilla/5.0 (compatible; aiHitBot-DM/2.0.2 +http://www.aihit.com)
Mozilla/5.0 (compatible; aiHitBot/1.0-DS; +http://www.aihit.com/)
Mozilla/5.0 (compatible; aiHitBot/1.0; +http://www.aihit.com/)
Mozilla/5.0 (compatible; aiHitBot/1.1; +http://www.aihit.com/)
Mozilla/5.0 (compatible; aiHitBot/2.7; +http://www.aihit.com/)
Mozilla/5.0 (compatible; aiHitBot/2.8; +http://endb-consolidated.aihit.com/)
Mozilla/5.0 (compatible; aiHitBot/2.9; +http://www.aihitdata.com/about)
Mozilla/5.0 (compatible; aiHitBot/2.9; +https://www.aihitdata.com/about)
Mozilla/5.0 (compatible; akula/12.0rc-2; +http://k311.fd.cvut.cz/)
Mozilla/5.0 (compatible; akula/k311; +http://k311.fd.cvut.cz/)
Mozilla/5.0 (compatible; alexa site audit/1.0; +http://www.alexa.com/help/webmasters; )
Mozilla/5.0 (compatible; alexa site audit/1.0; +http://www.alexa.com/help/webmasters; no-reply@alexa.com)
Mozilla/5.0 (compatible; alexa site audit/1.0; +http://www.alexa.com/help/webmasters; siteaudit@alexa.com)
Mozilla/5.0 (compatible; alexa site audit/1.0; http://www.alexa.com/help/webmasters; )
Mozilla/5.0 (compatible; archive-com.com/1.1; +http://archive-com.com/bot)
Mozilla/5.0 (compatible; archive.bibalex.org_bot; +http://www.test.com)
Mozilla/5.0 (compatible; archive.org_bot +http://archive.org/details/archive.org_bot)
Mozilla/5.0 (compatible; archive.org_bot +http://www.archive.org/details/archive.org_bot)
Mozilla/5.0 (compatible; archive.org_bot/3.1.2 +http://natlib.govt.nz/about-us/current-initiatives/web-harvest-2012)
Mozilla/5.0 (compatible; archive.org_bot/3.3.0 +http://pandora.nla.gov.au/crawl.html)
Mozilla/5.0 (compatible; archive.org_bot/3.3.0 +http://www.archive.org/details/archive.org_bot)
Mozilla/5.0 (compatible; archive.org_bot/heritrix-1.15.4 +http://www.archive.org)
Mozilla/5.0 (compatible; archive.org_bot; Archive-It; +http://archive-it.org/files/site-owners.html)
Mozilla/5.0 (compatible; archive.org_bot; Wayback Machine Live Record; +http://archive.org/details/archive.org_bot)
Mozilla/5.0 (compatible; archive_crawler/3.0.0-SNAPSHOT-20091205.013431 http://www.archive.org/details/archive_crawler)
Mozilla/5.0 (compatible; ayna-crawler http://www.ayna.com)
Mozilla/5.0 (compatible; bingbot/2.0;  http://www.bing.com/bingbot.htm)
Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm
Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)
Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm) SitemapProbe
Mozilla/5.0 (compatible; bixolabs/1.0; +http://bixolabs.com/crawler/general; crawler@bixolabs.com)
Mozilla/5.0 (compatible; bixolabs/1.0; +http://bixolabs.com/crawler/general; crawler@mail.bixolabs.com)
Mozilla/5.0 (compatible; bnf.fr_bot; +http://bibnum.bnf.fr/robot/bnf.html)
Mozilla/5.0 (compatible; bnf.fr_bot; +http://www.bnf.fr/fr/outils/a.dl_web_capture_robot.html)
Mozilla/5.0 (compatible; botify; http://botify.com)
Mozilla/5.0 (compatible; ca-crawler/1.0)
Mozilla/5.0 (compatible; clumboot; +http://localhost/clumpit/bot.php)
Mozilla/5.0 (compatible; coccoc/1.0; +http://help.coccoc.com/)
Mozilla/5.0 (compatible; coccoc/1.0; +http://help.coccoc.com/searchengine)
Mozilla/5.0 (compatible; coccocbot-ads/1.0; +http://help.coccoc.com/searchengine)
Mozilla/5.0 (compatible; coccocbot-fast/1.0; +http://help.coccoc.com/searchengine)
Mozilla/5.0 (compatible; coccocbot-image/1.0; +http://help.coccoc.com/searchengine)
Mozilla/5.0 (compatible; coccocbot-shopping/1.0; +http://help.coccoc.com/searchengine)
Mozilla/5.0 (compatible; coccocbot-web/1.0; +http://help.coccoc.com/searchengine)
Mozilla/5.0 (compatible; coccocbot/1.0; +http://help.coccoc.com/searchengine)
Mozilla/5.0 (compatible; discobot/1.0; +http://discoveryengine.com/discobot.html)
Mozilla/5.0 (compatible; discobot/2.0; +http://discoveryengine.com/discobot.html)
Mozilla/5.0 (compatible; discoverybot/2.0; +http://discoveryengine.com/discoverybot.html)
Mozilla/5.0 (compatible; dlcbot/0.1; +http://www.drlinkcheck.com/)
Mozilla/5.0 (compatible; dotSemantic/1.0; +http://www.dotsemantic.org)
Mozilla/5.0 (compatible; electricmonk/3.1.1 +https://www.duedil.com/our-crawler/)
Mozilla/5.0 (compatible; electricmonk/3.2.0 +https://www.duedil.com/our-crawler/)
Mozilla/5.0 (compatible; emefgebot/beta; +http://emefge.de/bot.html)
Mozilla/5.0 (compatible; epicbot; +http://www.epictions.com/epicbot)
Mozilla/5.0 (compatible; eright/1.0; +bot@eright.com)
Mozilla/5.0 (compatible; etoolsbot/1.0; +https://etools.io/bots/)
Mozilla/5.0 (compatible; evc-batch/2.0)
Mozilla/5.0 (compatible; evc-batch/2.0.20160608212921)
Mozilla/5.0 (compatible; evc-batch/2.0.20160708184126)
Mozilla/5.0 (compatible; evc-batch/2.0.20160902205649)
Mozilla/5.0 (compatible; evc-batch/2.0.20161017175820)
Mozilla/5.0 (compatible; evc-batch/2.0.20161026010401)
Mozilla/5.0 (compatible; evc-batch/2.0.20161122223033)
Mozilla/5.0 (compatible; evc-batch/2.0.20161220184139)
Mozilla/5.0 (compatible; evc-batch/2.0.20170110125844)
Mozilla/5.0 (compatible; firmilybot/0.3; +http://www.firmily.com/bot.php
Mozilla/5.0 (compatible; forensiq; +http://www.forensiq.com)
Mozilla/5.0 (compatible; fr-crawler/1.1)
Mozilla/5.0 (compatible; gimme60bot/1.0 ; +http://gimme60.com)
Mozilla/5.0 (compatible; gofind; +http://govid.mobi/bot.php)
Mozilla/5.0 (compatible; grapeFX/0.9; crawler@grapeshot.co.uk
Mozilla/5.0 (compatible; heritrix/1.12.1 +http://www.webarchiv.cz)
Mozilla/5.0 (compatible; heritrix/1.12.1b +http://netarkivet.dk/website/info.html)
Mozilla/5.0 (compatible; heritrix/1.14.2 +http://rjpower.org)
Mozilla/5.0 (compatible; heritrix/1.14.2 +http://www.webarchiv.cz)
Mozilla/5.0 (compatible; heritrix/1.14.3 +http://archive.org)
Mozilla/5.0 (compatible; heritrix/1.14.3 +http://www.accelobot.com)
Mozilla/5.0 (compatible; heritrix/1.14.3 +http://www.webarchiv.cz)
Mozilla/5.0 (compatible; heritrix/1.14.3.r6601 +http://www.buddybuzz.net/yptrino)
Mozilla/5.0 (compatible; heritrix/1.14.4 +http://parsijoo.ir)
Mozilla/5.0 (compatible; heritrix/1.14.4 +http://webarhive.nlc.gov.cn)
Mozilla/5.0 (compatible; heritrix/1.14.4 +http://www.dla-marbach.de)
Mozilla/5.0 (compatible; heritrix/1.14.4 +http://www.exif-search.com)
Mozilla/5.0 (compatible; heritrix/2.0.2 +http://aihit.com)
Mozilla/5.0 (compatible; heritrix/2.0.2 +http://seekda.com)
Mozilla/5.0 (compatible; heritrix/3.0.0-SNAPSHOT-20091120.021634 +http://crawler.archive.org)
Mozilla/5.0 (compatible; heritrix/3.1.0-RC1 +http://boston.lti.cs.cmu.edu/crawler_12/)
Mozilla/5.0 (compatible; heritrix/3.1.1 +http://places.tomtom.com/crawlerinfo)
Mozilla/5.0 (compatible; heritrix/3.1.1 +http://www.baidu.com)
Mozilla/5.0 (compatible; heritrix/3.1.1 +http://www.mixdata.com)
Mozilla/5.0 (compatible; heritrix/3.1.1 +http://www.run4dom.com)
Mozilla/5.0 (compatible; heritrix/3.1.1-SNAPSHOT-20120116.200628 +http://www.archive.org/details/archive.org_bot)
Mozilla/5.0 (compatible; heritrix/3.1.1; UniLeipzigASV +http://corpora.informatik.uni-leipzig.de/crawler_faq.html)
Mozilla/5.0 (compatible; heritrix/3.1.2-SNAPSHOT-20130207.001528 +http://webarchiv.cz/kontakty/)
Mozilla/5.0 (compatible; heritrix/3.1.2-SNAPSHOT-20131029-0036 +http://archive.org/details/archive.org_bot)
Mozilla/5.0 (compatible; heritrix/3.2.0 +http://www.crim.ca)
Mozilla/5.0 (compatible; heritrix/3.2.0 +http://www.exif-search.com)
Mozilla/5.0 (compatible; heritrix/3.2.0 +http://www.mixdata.com)
Mozilla/5.0 (compatible; heritrix/3.3.0-SNAPSHOT-20140702-2247 +http://archive.org/details/archive.org_bot)
Mozilla/5.0 (compatible; heritrix/3.3.0-SNAPSHOT-20160309-0050; UniLeipzigASV +http://corpora.informatik.uni-leipzig.de/crawler_faq.html)
Mozilla/5.0 (compatible; historious/2.0; +http://historio.us/)
Mozilla/5.0 (compatible; houzzbot; +http://www.houzz.com/)
Mozilla/5.0 (compatible; htrix/1.14.4 ++https://webarchiv.onb.ac.at/robot.html)
Mozilla/5.0 (compatible; http://alyze.info)
Mozilla/5.0 (compatible; hypestat/1.0; +http://www.hypestat.com/bot)
Mozilla/5.0 (compatible; iThemes Sync/1.0; +http://ithemes.com/sync)
Mozilla/5.0 (compatible; iThemes Sync/1.0; ITX WP-Admin API/1.0; +http://ithemes.com/sync)
Mozilla/5.0 (compatible; idmarch Automatic.beta/1.2; +http://www.idmarch.org/bot.html)
Mozilla/5.0 (compatible; idmarch Automatic.beta/1.3; +http://www.idmarch.org/bot.html)
Mozilla/5.0 (compatible; idmarch Automatic.beta/1.4; +http://www.idmarch.org/bot.html)
Mozilla/5.0 (compatible; image.coccoc/1.0; +http://help.coccoc.com/)
Mozilla/5.0 (compatible; imagecoccoc/1.0; +http://help.coccoc.com/)
Mozilla/5.0 (compatible; imagecoccoc/1.0; +http://help.coccoc.com/searchengine)
Mozilla/5.0 (compatible; imbot/0.1 +http://internetmemory.org/en/
Mozilla/5.0 (compatible; imbot/0.1 +http://internetmemory.org/en/)
Mozilla/5.0 (compatible; imrbot/1.10.8 +http://www.mignify.com)
Mozilla/5.0 (compatible; informatique at mixdata dot com/0.6; +http://t.co/GSRLLKex24; informatique at mixdata dot com)
Mozilla/5.0 (compatible; inoreader.com-like FeedFetcher-Google)
Mozilla/5.0 (compatible; inoreader.com; 2 subscribers)
Mozilla/5.0 (compatible; ips-agent)
Mozilla/5.0 (compatible; janforman.com/2.4; +http://janforman.com/)
Mozilla/5.0 (compatible; janforman.com/2.5; +http://janforman.com/)
Mozilla/5.0 (compatible; jpg-newsbot/2.0; +http://vipnytt.no/bot.html)
Mozilla/5.0 (compatible; kazbtbot/0.1; +http://kazbt.com/)
Mozilla/5.0 (compatible; kgbody/2.0; +http://kids.goo.ne.jp/tool/kgbody.php)
Mozilla/5.0 (compatible; kulturarw3 +http://www.kb.se/om/projekt/Svenska-webbsidor---Kulturarw3/)
Mozilla/5.0 (compatible; lemurwebcrawler admin@lemurproject.org; +http://boston.lti.cs.cmu.edu/crawler_12/)
Mozilla/5.0 (compatible; lincobot/0.3; http://www.linksco.com/about/robot.html) AppEngine-Google; (+http://code.google.com/appengine)
Mozilla/5.0 (compatible; linkdexbot/2.0; +http://www.linkdex.com/about/bots/)
Mozilla/5.0 (compatible; linkdexbot/2.0; +http://www.linkdex.com/bots/)
Mozilla/5.0 (compatible; linkdexbot/2.1; +http://www.linkdex.com/about/bots/)
Mozilla/5.0 (compatible; linkdexbot/2.1; +http://www.linkdex.com/bots/)
Mozilla/5.0 (compatible; linkdexbot/2.2; +http://www.linkdex.com/bots/)
Mozilla/5.0 (compatible; loc-crawl/1.10.1 +http://www.google.com)
Mozilla/5.0 (compatible; loc-crawler/0.11.0 +http://loc.gov),gzip(gfe),gzip(gfe),gzip(gfe)
Mozilla/5.0 (compatible; localsearch-web/2.0; +http://www.localsearch.ch/en/legal-notice)
Mozilla/5.0 (compatible; ltbot/0.3.6 +http://www.kdsl.tu-darmstadt.de/de/kdsl/research-program/crawling-and-semantic-structuring/)
Mozilla/5.0 (compatible; ltbot/3.2.0.10 +http://www.kdsl.tu-darmstadt.de/de/kdsl/research-program/crawling-and-semantic-structuring/)
Mozilla/5.0 (compatible; meanpathbot/1.0; +http://www.meanpath.com/meanpathbot.html)
Mozilla/5.0 (compatible; meg/0.2; +https://github.com/tomnomnom/meg)
Mozilla/5.0 (compatible; memoryBot/1.20.199 +http://internetmemory.org/en/)
Mozilla/5.0 (compatible; memoryBot/1.20.210 +http://internetmemory.org/en/)
Mozilla/5.0 (compatible; memoryBot/1.20.235 +http://internetmemory.org/en/)
Mozilla/5.0 (compatible; memoryBot/1.20.241 +http://internetmemory.org/en/)
Mozilla/5.0 (compatible; memoryBot/1.20.243 +http://internetmemory.org/en/)
Mozilla/5.0 (compatible; memoryBot/1.20.247 +http://internetmemory.org/en/)
Mozilla/5.0 (compatible; memoryBot/1.20.268 +http://internetmemory.org/en/)
Mozilla/5.0 (compatible; memoryBot/1.20.275 +http://internetmemory.org/en/)
Mozilla/5.0 (compatible; memoryBot/1.20.289 +http://internetmemory.org/en/)
Mozilla/5.0 (compatible; memoryBot/1.20.311 +http://internetmemory.org/en/)
Mozilla/5.0 (compatible; memoryBot/1.21.14 +http://mignify.com/bot.html)
Mozilla/5.0 (compatible; memoryBot/1.21.16 +http://internetmemory.org/en/)
Mozilla/5.0 (compatible; memoryBot/1.21.23 +http://internetmemory.org/en/)
Mozilla/5.0 (compatible; memoryBot/1.21.24 +http://internetmemory.org/en/)
Mozilla/5.0 (compatible; memoryBot/1.21.46 +http://internetmemory.org/en/)
Mozilla/5.0 (compatible; memoryBot/1.24.11 +http://internetmemory.org/en/)
Mozilla/5.0 (compatible; memoryBot/1.24.54 +http://internetmemory.org/en/)
Mozilla/5.0 (compatible; memoryBot/1.24.61 +http://internetmemory.org/en/)
Mozilla/5.0 (compatible; memorybot/1.20.207 +http://mignify.com/bot.html)
Mozilla/5.0 (compatible; memorybot/1.20.71 +http://archivethe.net/en/index.php/about/internet_memory1 on behalf of DNB)
Mozilla/5.0 (compatible; memorybot/1.21.14 +http://mignify.com/bot.html)
Mozilla/5.0 (compatible; memorybot/1.24.12 +https://internetmemory.net/en/bots)
Mozilla/5.0 (compatible; metager2-verification-bot; +http://metager2.de/technology.php)
Mozilla/5.0 (compatible; monitis - premium monitoring service; http://www.monitis.com)
Mozilla/5.0 (compatible; musobot/1.0; info@muso.com; +http://www.muso.com)
Mozilla/5.0 (compatible; mxbot/1.0; http://www.chainn.com/mxbot.html)
Mozilla/5.0 (compatible; nbot/2.0)
Mozilla/5.0 (compatible; oBot/2.3.1; +http://filterdb.iss.net/crawler/)
Mozilla/5.0 (compatible; oBot/2.3.1; +http://www-935.ibm.com/services/us/index.wss/detail/iss/a1029077?cntxt=a1027244)
Mozilla/5.0 (compatible; oBot/2.3.1; http://filterdb.iss.net/crawler/)
Mozilla/5.0 (compatible; oBot/2.3.1; http://www.xforce-security.com/crawler/)
Mozilla/5.0 (compatible; officestorebot/1.0; +https://aka.ms/officestorebot)
Mozilla/5.0 (compatible; online-webceo-bot/1.0; +http://online.webceo.com)
Mozilla/5.0 (compatible; openstat.ru/Bot)
Mozilla/5.0 (compatible; parse.ly scraper/0.14; +http://parsely.com)
Mozilla/5.0 (compatible; parsijoo-bot; +http://www.parsijoo.ir/; ehsan.mousakazemi@gmail.com)
Mozilla/5.0 (compatible; parsijoo-bot; +http://www.parsijoo.ir/; ehsanmousa@parsijoo.ir)
Mozilla/5.0 (compatible; parsijoo-update-crawler; +http://www.parsijoo.ir/; ehsan.mousakazemi@gmail.com)
Mozilla/5.0 (compatible; parsijoo-update-crawler; +http://www.parsijoo.ir/; ehsanmousa@parsijoo.ir)
Mozilla/5.0 (compatible; parsijoo; +http://www.parsijoo.ir/; ehsan.mousakazemi@gmail.com)
Mozilla/5.0 (compatible; pdffillerbot/1.0; +http://www.pdffiller.com/en/support.htm)
Mozilla/5.0 (compatible; phpservermon/3.1.1; +http://www.phpservermonitor.org)
Mozilla/5.0 (compatible; pingbot/2.0; +http://www.pingdom.com/)
Mozilla/5.0 (compatible; pmoz.info ODP link checker; +http://pmoz.info/doc/botinfo.htm)
Mozilla/5.0 (compatible; proximic; +http://www.proximic.com)
Mozilla/5.0 (compatible; proximic; +http://www.proximic.com/info/spider.php)
Mozilla/5.0 (compatible; proximic; https://www.comscore.com/Web-Crawler)
Mozilla/5.0 (compatible; ptd-crawler; +http://bixolabs.com/crawler/ptd/; crawler@bixolabs.com)
Mozilla/5.0 (compatible; ptd-crawler; http://bixolabs.com/crawler/ptd/; crawler@bixolabs.com)
Mozilla/5.0 (compatible; pub-crawler; +http://wiki.github.com/bixo/bixo/bixocrawler; bixo-dev@yahoogroups.com)
Mozilla/5.0 (compatible; redditbot/1.0; +http://www.reddit.com/feedback)
Mozilla/5.0 (compatible; rogerBot/1.0; UrlCrawler; http://www.seomoz.org/dp/rogerbot)
Mozilla/5.0 (compatible; sReleaseBot/6.1; +http://www.srelease.com/)
Mozilla/5.0 (compatible; scanbot/1.0; +http://dazzlepod.com/ip/)
Mozilla/5.0 (compatible; seo-audit-check-bot/1.0)
Mozilla/5.0 (compatible; seolyt/1.1; +https://seolyt.com)
Mozilla/5.0 (compatible; seoscanners.net/1; +spider@seoscanners.net)
Mozilla/5.0 (compatible; seplinkbot/1.0 )
Mozilla/5.0 (compatible; sgbot v0.01a, sgcrawlerbot@gmail.com)
Mozilla/5.0 (compatible; socialbm_bot/1.0; +http://spider.socialbm.net)
Mozilla/5.0 (compatible; socketcrawler; http://nlp.fi.muni.cz/projects/biwec/)
Mozilla/5.0 (compatible; spbot/1.0; +http://www.seoprofiler.com/bot/ )
Mozilla/5.0 (compatible; spbot/1.1; +http://www.seoprofiler.com/bot/ )
Mozilla/5.0 (compatible; spbot/1.2; +http://www.seoprofiler.com/bot/ )
Mozilla/5.0 (compatible; spbot/2.0.1; +http://www.seoprofiler.com/bot/ )
Mozilla/5.0 (compatible; spbot/2.0.2; +http://www.seoprofiler.com/bot/ )
Mozilla/5.0 (compatible; spbot/2.0.3; +http://www.seoprofiler.com/bot/ )
Mozilla/5.0 (compatible; spbot/2.0.4; +http://www.seoprofiler.com/bot )
Mozilla/5.0 (compatible; spbot/2.0; +http://www.seoprofiler.com/bot/ )
Mozilla/5.0 (compatible; spbot/2.1; +http://www.seoprofiler.com/bot )
Mozilla/5.0 (compatible; spbot/3.0; +http://www.seoprofiler.com/bot )
Mozilla/5.0 (compatible; spbot/3.1; +http://www.seoprofiler.com/bot )
Mozilla/5.0 (compatible; spbot/4.0.1; +http://www.seoprofiler.com/bot )
Mozilla/5.0 (compatible; spbot/4.0.2; +http://www.seoprofiler.com/bot )
Mozilla/5.0 (compatible; spbot/4.0.3; +http://www.seoprofiler.com/bot )
Mozilla/5.0 (compatible; spbot/4.0.4; +http://www.seoprofiler.com/bot )
Mozilla/5.0 (compatible; spbot/4.0.5; +http://www.seoprofiler.com/bot )
Mozilla/5.0 (compatible; spbot/4.0.6; +http://www.seoprofiler.com/bot )
Mozilla/5.0 (compatible; spbot/4.0.7; +http://OpenLinkProfiler.org/bot )
Mozilla/5.0 (compatible; spbot/4.0.7; +https://www.seoprofiler.com/bot )
Mozilla/5.0 (compatible; spbot/4.0.8; +http://OpenLinkProfiler.org/bot )
Mozilla/5.0 (compatible; spbot/4.0.9; +http://OpenLinkProfiler.org/bot )
Mozilla/5.0 (compatible; spbot/4.0; +http://www.seoprofiler.com/bot )
Mozilla/5.0 (compatible; spbot/4.0a; +http://www.seoprofiler.com/bot )
Mozilla/5.0 (compatible; spbot/4.0b; +http://www.seoprofiler.com/bot )
Mozilla/5.0 (compatible; spbot/4.1.0; +http://OpenLinkProfiler.org/bot )
Mozilla/5.0 (compatible; spbot/4.2.0; +http://OpenLinkProfiler.org/bot )
Mozilla/5.0 (compatible; spbot/4.3.0; +http://OpenLinkProfiler.org/bot )
Mozilla/5.0 (compatible; spbot/4.4.0; +http://OpenLinkProfiler.org/bot )
Mozilla/5.0 (compatible; spbot/4.4.1; +http://OpenLinkProfiler.org/bot )
Mozilla/5.0 (compatible; spbot/4.4.2; +http://OpenLinkProfiler.org/bot )
Mozilla/5.0 (compatible; spbot/5.0.1; +http://OpenLinkProfiler.org/bot )
Mozilla/5.0 (compatible; spbot/5.0.2; +http://OpenLinkProfiler.org/bot )
Mozilla/5.0 (compatible; spbot/5.0.3; +http://OpenLinkProfiler.org/bot )
Mozilla/5.0 (compatible; spbot/5.0; +http://OpenLinkProfiler.org/bot )
Mozilla/5.0 (compatible; special_archiver/3.1.1 +http://www.archive.org/details/archive.org_bot)
Mozilla/5.0 (compatible; special_archiver/3.2.0 +http://www.loc.gov/webarchiving/notice_to_webmasters.html)
Mozilla/5.0 (compatible; special_archiver/3.3.0 +http://www.loc.gov/webarchiving/notice_to_webmasters.html)
Mozilla/5.0 (compatible; special_archiver/3.3.0 bot@archive.org +https://archive.org/details/archive.org_bot)
Mozilla/5.0 (compatible; startmebot/1.0; +http://www.start.me/bot)
Mozilla/5.0 (compatible; startmebot/1.0; +https://start.me/bot)
Mozilla/5.0 (compatible; suggybot v0.01a, http://blog.suggy.com/was-ist-suggy/suggy-webcrawler/)
Mozilla/5.0 (compatible; sukibot_heritrix/3.1.1 +http://suki.ling.helsinki.fi/eng/webmasters.html)
Mozilla/5.0 (compatible; summers;)/Nutch-1.7
Mozilla/5.0 (compatible; tchelebi/1.0; +http://tchelebi.io)
Mozilla/5.0 (compatible; theoldreader.com)
Mozilla/5.0 (compatible; theoldreader.com; 1 subscribers; feed-id=51413de5bd9279ea5002d75b)
Mozilla/5.0 (compatible; theoldreader.com; 1 subscribers; feed-id=aaa)
Mozilla/5.0 (compatible; tldstat.com/Bot; +http://tldstat.com/bot.html)
Mozilla/5.0 (compatible; tracemyfile/1.0)
Mozilla/5.0 (compatible; tracemyfile/1.0; +bot@tracemyfile.com)
Mozilla/5.0 (compatible; trovitBot 1.0; +http://www.trovit.com/bot.html)
Mozilla/5.0 (compatible; uCrawlr/1.0 ; +https://blog.ucoz.ru/upolicy)
Mozilla/5.0 (compatible; uMBot-FC/1.0; mailto: crawling@ubermetrics-technologies.com)
Mozilla/5.0 (compatible; uMBot-LN/1.0; mailto: crawling@ubermetrics-technologies.com)
Mozilla/5.0 (compatible; um-LN/1.0; mailto: techinfo@ubermetrics-technologies.com)
Mozilla/5.0 (compatible; value/1.1; +http://www.aihit.com/)
Mozilla/5.0 (compatible; vebidoobot/1.0; +https://blog.vebidoo.de/vebidoobot/
Mozilla/5.0 (compatible; vkShare; +http://vk.com/dev/Share)
Mozilla/5.0 (compatible; waybackarchive.org/1.0; +spider@waybackarchive.org)
Mozilla/5.0 (compatible; websays; +http://wiki.github.com/bixo/bixo/bixocrawler; bixo-dev@yahoogroups.com)
Mozilla/5.0 (compatible; wmtips.com/1.0; +http://www.wmtips.com/tools/)
Mozilla/5.0 (compatible; woobot/2.0; +https://www.woorank.com/bot)
Mozilla/5.0 (compatible; woorankreview/2.0; +https://www.woorank.com/)
Mozilla/5.0 (compatible; worio bot heritrix/1.10.0 +http://worio.com)
Mozilla/5.0 (compatible; woriobot +http://worio.com)
Mozilla/5.0 (compatible; woriobot support [at] zite [dot] com +http://zite.com)
Mozilla/5.0 (compatible; www.monitor.us - free monitoring service; http://www.monitor.us)
Mozilla/5.0 (compatible; yoozBot-2.2; http://yooz.ir; info@yooz.ir)
Mozilla/5.0 (compatible; zenback bot; powered by logly +http://corp.logly.co.jp/)
Mozilla/5.0 (compatible; zitebot support [at] zite [dot] com +http://zite.com)
Mozilla/5.0 (compatible; zyklop; +http://www.seoratio.de/zyklop-crawler/)
Mozilla/5.0 (compatible;+ParchBot/1.0;++http://www.parchmenthill.com/search.htm)
Mozilla/5.0 (compatible;AspiegelBot)
Mozilla/5.0 (compatible;Cloudflare-Healthchecks/1.0;+https://www.cloudflare.com/; healthcheck-id: 755842180c7ae904)
Mozilla/5.0 (compatible;FindITAnswersbot/1.0;+http://search.it-influentials.com/bot.htm)
Mozilla/5.0 (compatible;PetalBot;+https://webmaster.petalsearch.com/site/petalbot)
Mozilla/5.0 (compatible;WI Job Roboter Spider Version 3;+http://www.webintegration.at)
Mozilla/5.0 (compatible;YodaoBot-Image/1.0;http://www.youdao.com/help/webmaster/spider/;)
Mozilla/5.0 (compatible;acapbot/0.1)
Mozilla/5.0 (compatible;acapbot/0.1.;treat like Googlebot)
Mozilla/5.0 (compatible;acapbot/0.1;treat like Googlebot)
Mozilla/5.0 (compatible;contxbot/1.0)
Mozilla/5.0 (compatible;picmole/1.0 +http://www.picmole.com)
Mozilla/5.0 (en-US) AppleWebKit/537.36 (KHTML, like Gecko; Google-Assess) Chrome/34.0.1847.116 Safari/537.36
Mozilla/5.0 (en-us) AppleWebKit/525.13 (KHTML, like Gecko; Google Web Preview) Version/3.1 Safari/525.13
Mozilla/5.0 (en-us) AppleWebKit/534.14 (KHTML, like Gecko; Google Wireless Transcoder) Chrome/9.0.597 Safari/534.14
Mozilla/5.0 (en-us) AppleWebKit/537.36(KHTML like Gecko; Google-Adwords-DisplayAds-WebRender;) Chrome/27.0.1453Safari/537.36
Mozilla/5.0 (en-us) AppleWebKit/537.36(KHTML, like Gecko; Google-Adwords-DisplayAds-WebRender;) Chrome/27.0.1453Safari/537.36
Mozilla/5.0 (en-us) AppleWebKit/537.36(KHTML, like Gecko; Google-Adwords-DisplayAds-WebRender;) Chrome/41.0.2272.118Safari/537.36
Mozilla/5.0 (iPhone; CPU iPhone OS 10_0_1 like Mac OS X) AppleWebKit/602.1.50 (KHTML, like Gecko) Version/10.0 Mobile/14A403 Safari/602.1 DareBoost
Mozilla/5.0 (iPhone; CPU iPhone OS 11_0 like Mac OS X) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/40.0.3754.1902 Mobile Safari/537.36; Bytespider
Mozilla/5.0 (iPhone; CPU iPhone OS 11_0 like Mac OS X) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.4454.1745 Mobile Safari/537.36; Bytespider
Mozilla/5.0 (iPhone; CPU iPhone OS 11_0 like Mac OS X) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/43.0.7597.1164 Mobile Safari/537.36; Bytespider;bytespider@bytedance.com
Mozilla/5.0 (iPhone; CPU iPhone OS 11_0 like Mac OS X) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/46.0.2988.1545 Mobile Safari/537.36; Bytespider
Mozilla/5.0 (iPhone; CPU iPhone OS 11_0 like Mac OS X) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.4141.1682 Mobile Safari/537.36; Bytespider
Mozilla/5.0 (iPhone; CPU iPhone OS 11_0 like Mac OS X) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.3478.1649 Mobile Safari/537.36; Bytespider
Mozilla/5.0 (iPhone; CPU iPhone OS 11_0 like Mac OS X) AppleWebKit/604.1.38 (KHTML, like Gecko) Version/11.0 Mobile/15A372 Safari/604.1 (compatible; woorankreview/2.0; +https://www.woorank.com/)
Mozilla/5.0 (iPhone; CPU iPhone OS 12_0 like Mac OS X) adbeat.com/policy AppleWebKit/605.1.15 (KHTML, like Gecko) CriOS/69.0.3497.105 Mobile/15E148 Safari/605.1
Mozilla/5.0 (iPhone; CPU iPhone OS 5_0_1 like Mac OS X) (compatible; Yeti-Mobile/0.1; +http://help.naver.com/robots/)
Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 (KHTML, like Gecko)                 Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; SMTBot/1.0; +http://www.similartech.com/smtbot)
Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 (KHTML, like Gecko) Version/6.0 Mobile/10A5376e Safari/8536.25 ( compatible; CloudServerMarketSpider/1.0; +http://cloudservermarket.com/spider.html )
Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 (KHTML, like Gecko) Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; Applebot/0.3; +http://www.apple.com/go/applebot)
Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 (KHTML, like Gecko) Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; Googlebot-Mobile/2.1; +http://www.google.com/bot.html)
Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 (KHTML, like Gecko) Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
Mozilla/5.0 (iPhone; CPU iPhone OS 6_0_1 like Mac OS X) AppleWebKit/537.36 (KHTML, like Gecko; Google Page Speed Insights) Version/6.0 Mobile/10A525 Safari/8536.25
Mozilla/5.0 (iPhone; CPU iPhone OS 6_0_1 like Mac OS X) AppleWebKit/537.36 (KHTML, like Gecko; Google Page Speed Insights) Version/6.0 Mobile/10A525 Safari/8536.25 GoogleBot/2.1
Mozilla/5.0 (iPhone; CPU iPhone OS 7_0 like Mac OS X) AppleWebKit/537.51.1 (KHTML, like Gecko) Mobile/11A465 Twitter for iPhone BrandVerity/1.0 (http://www.brandverity.com/why-is-brandverity-visiting-me)
Mozilla/5.0 (iPhone; CPU iPhone OS 7_0 like Mac OS X) AppleWebKit/537.51.1 (KHTML, like Gecko) Version/7.0 Mobile/11A465 Safari/9537.53 (compatible; adidxbot/2.0;  http://www.bing.com/bingbot.htm)
Mozilla/5.0 (iPhone; CPU iPhone OS 7_0 like Mac OS X) AppleWebKit/537.51.1 (KHTML, like Gecko) Version/7.0 Mobile/11A465 Safari/9537.53 (compatible; adidxbot/2.0; +http://www.bing.com/bingbot.htm)
Mozilla/5.0 (iPhone; CPU iPhone OS 7_0 like Mac OS X) AppleWebKit/537.51.1 (KHTML, like Gecko) Version/7.0 Mobile/11A465 Safari/9537.53 (compatible; bingbot/2.0;  http://www.bing.com/bingbot.htm)
Mozilla/5.0 (iPhone; CPU iPhone OS 7_0 like Mac OS X) AppleWebKit/537.51.1 (KHTML, like Gecko) Version/7.0 Mobile/11A465 Safari/9537.53 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)
Mozilla/5.0 (iPhone; CPU iPhone OS 7_0 like Mac OS X) AppleWebKit/537.51.1 (KHTML, like Gecko) Version/7.0 Mobile/11A465 Safari/9537.53 BingPreview/1.0b
Mozilla/5.0 (iPhone; CPU iPhone OS 7_1 like Mac OS X) AppleWebKit (KHTML, like Gecko) Mobile (compatible; Yahoo Ad monitoring; https://help.yahoo.com/kb/yahoo-ad-monitoring-SLN24857.html)
Mozilla/5.0 (iPhone; CPU iPhone OS 7_1 like Mac OS X) AppleWebKit (KHTML, like Gecko) Mobile (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)
Mozilla/5.0 (iPhone; CPU iPhone OS 8_0 like Mac OS X) AppleWebKit/600.1.3 (KHTML, like Gecko) Version/8.0 Mobile/12A4345d Safari/600.1.4 moatbot
Mozilla/5.0 (iPhone; CPU iPhone OS 8_1 like Mac OS X) AppleWebKit/600.1.4 (KHTML, like Gecko) Version/8.0 Mobile/12B410 Safari/600.1.4 (Applebot/0.1; +http://www.apple.com/go/applebot)
Mozilla/5.0 (iPhone; CPU iPhone OS 8_1 like Mac OS X) AppleWebKit/600.1.4 (KHTML, like Gecko) Version/8.0 Mobile/12B411 Safari/600.1.4 (compatible; YandexBot/3.0; +http://yandex.com/bots)
Mozilla/5.0 (iPhone; CPU iPhone OS 8_1 like Mac OS X) AppleWebKit/600.1.4 (KHTML, like Gecko) Version/8.0 Mobile/12B411 Safari/600.1.4 (compatible; YandexMobileBot/3.0; +http://yandex.com/bots)
Mozilla/5.0 (iPhone; CPU iPhone OS 8_3 like Mac OS X) AppleWebKit/537.36 (KHTML, like Gecko) Version/8.0 Mobile/12F70 Safari/600.1.4 (compatible; Google Search Console)
Mozilla/5.0 (iPhone; CPU iPhone OS 8_3 like Mac OS X) AppleWebKit/537.36 (KHTML, like Gecko) Version/8.0 Mobile/12F70 Safari/600.1.4 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
Mozilla/5.0 (iPhone; CPU iPhone OS 8_3 like Mac OS X) AppleWebKit/537.36 (KHTML, like Gecko; Google Page Speed Insights) Version/8.0 Mobile/12F70 Safari/600.1.4
Mozilla/5.0 (iPhone; CPU iPhone OS 8_3 like Mac OS X) AppleWebKit/600.1.4 (KHTML, like Gecko) Version/8.0 Mobile/12F70 Safari/600.1. 4 (compatible; GrapeshotCrawler/2.0; +https://www.grapeshot.com/crawler/)
Mozilla/5.0 (iPhone; CPU iPhone OS 8_3 like Mac OS X) AppleWebKit/600.1.4 (KHTML, like Gecko) Version/8.0 Mobile/12F70 Safari/600.1.4 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
Mozilla/5.0 (iPhone; CPU iPhone OS 8_3 like Mac OS X) AppleWebKit/600.1.4 (KHTML, like Gecko) Version/8.0 Mobile/12F70 Safari/600.1.4 (compatible; Laserlikebot/0.1)
Mozilla/5.0 (iPhone; CPU iPhone OS 9_1 like Mac OS X) AppleWebKit/601.1.46 (KHTML, like Gecko) Version/9.0 Mobile/13B143 Safari/601.1 (compatible; AdsBot-Google-Mobile; +http://www.google.com/mobile/adsbot.html)
Mozilla/5.0 (iPhone; CPU iPhone OS 9_1 like Mac OS X) AppleWebKit/601.1.46 (KHTML, like Gecko) Version/9.0 Mobile/13B143 Safari/601.1 SeoSiteCheckup (https://seositecheckup.co
Mozilla/5.0 (iPhone; U; CPU iPhone OS 10_0 like Mac OS X; en-us) AppleWebKit/602.1.38 (KHTML, like Gecko) Version/10.0 Mobile/14A5297c Safari/602.1 (compatible; Mediapartners-Google/2.1; +http://www.google.com/bot.html)
Mozilla/5.0 (iPhone; U; CPU iPhone OS 3_0 like Mac OS X; en-us; EasouSpider; +http://www.easou.com/search/spider.html)
Mozilla/5.0 (iPhone; U; CPU iPhone OS 4_0_1 like Mac OS X; en-us) AppleWebKit/537.4 (KHTML, like Gecko; Google Page Speed Insights) Version/4.0.5 Mobile/8A306 Safari/6531.22.7
Mozilla/5.0 (iPhone; U; CPU iPhone OS 4_1 like Mac OS X; en-us) AppleWebKit/532.9 (KHTML, like Gecko) Version/4.0.5 Mobile/8B117 Safari/6531.22.7 (compatible; Googlebot-Mobile/2.1; +http://www.google.com/bot.html)
Mozilla/5.0 (iPhone; U; CPU iPhone OS 4_1 like Mac OS X; en-us) AppleWebKit/532.9 (KHTML, like Gecko) Version/4.0.5 Mobile/8B117 Safari/6531.22.7 (compatible; Mediapartners-Google/2.1; +http://www.google.com/bot.html)
Mozilla/5.0 (iPhone; U; CPU iPhone OS 4_1 like Mac OS X; en-us) AppleWebKit/532.9 (KHTML, like Gecko) Version/4.0.5 Mobile/8B117 Safari/6531.22.7 (compatible; linkdexbot-mobile/2.1; +http://www.linkdex.com/about/bots/)
Mozilla/5.0 (iPhone; U; CPU iPhone OS 4_3 like Mac OS X; en-GB) adbeat.com/policy AppleWebKit/533.17.9 (KHTML, like Gecko) Version/5.0.2 Mobile/8F190 Safari/6533.18.5
Mozilla/5.0 (iPhone; U; CPU iPhone OS 4_3 like Mac OS X; en-US) adbeat.com/policy AppleWebKit/533.17.9 (KHTML, like Gecko) Version/5.0.2 Mobile/8F190 Safari/6533.18.5
Mozilla/5.0 (iPhone; Y!J-BRY/YATSH crawler; http://help.yahoo.co.jp/help/jp/search/indexing/indexing-15.html)
Mozilla/5.0 (lbot/2.6.4) lbot@mailme.not
Mozilla/5.0 (nomore404.com robot/1.1; +https://nomore404.com/)
Mozilla/5.0 (seoanalyzer; compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)
Mozilla/5.0 AppleWebKit (compatible; s4a-probe/1.0; +https://www.seo4ajax.com/webscraper)
Mozilla/5.0 AppleWebKit (compatible; s4a/1.0; +https://www.seo4ajax.com/webscraper)
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.118 Safari/537.36 (compatible; Google-Shopping-Quality +http://www.google.com/merchants/tos/extend/DE/tos.html)
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.103 Safari/537.36 (Refindbot/1.0)
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; Google Web Preview Analytics) Chrome/27.0.1453 Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Googlebot/2.1; +http://www.google.com/bot.html) Safari/537.36
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm) Safari/537.36
Mozilla/5.0 AppleWebKit/537.4 (KHTML, like Gecko; compatible; Googlebot/2.1; +http://www.google.com/bot.html) Safari/537.4
Mozilla/5.0 AppleWebkit/537.36 (KHTML, like Gecko) Trident/7.0; rv:11.0, Chrome/42.0.2311.135, Edge/12.10136, http://www.shrinktheweb.com/, Webshot/0.9
Mozilla/5.0 CommonCrawler Node MNATI3RZFBD6ZTQNRVQGXBDCQOEXIA4CIDEUKNYZDF3MJSH5G7BCQ5EMADE5TLF.MHCZ3LLTWGHIHMYLM.YFBJBHE6R2YQTHOTJ3N5BGFP457JRX4AJRXGAJYGDAANRY24.cdn0.common.crawl.zone
Mozilla/5.0 GurujiBot/1.0 ( http://www.guruji.com/en/WebmasterFAQ.html)
Mozilla/5.0 Moreover/5.1 (+http://www.moreover.com)
Mozilla/5.0 Moreover/5.1 (+http://www.moreover.com; webmaster@moreover.com)
Mozilla/5.0 PhantomJS (compatible; Seznam screenshot-generator 2.1; +http://fulltext.sblog.cz/screenshot/)
Mozilla/5.0 SeznamEmailProxy/2.0.174
Mozilla/5.0 compatible; yelpspider/yelpspider-1.0 (Crawlerbot run by Yelp Inc; yelpbot at yelp dot com)
Mozilla/5.0 eCairn-Grabber/1.0 (+http://ecairn.com/grabber)
Mozilla/5.0 zgrab/0.x
Mozilla/5.0(compatible; Sosospider/2.0; +http://help.soso.com/webspider.htm)
Mozilla/5.0(compatible;Sosospider/2.0;+http://help.soso.com/webspider.htm)
Mozilla/5.0+(compatible; Cula/2.0; https://cula.io/)
Mozilla/5.0+(compatible; Monitority/1.0; http://www.monitority.com/)
Mozilla/5.0+(compatible; MxToolbox/Beta7; http://www.mxtoolbox.com/)
Mozilla/5.0+(compatible; RevIP.info site analyzer v4.00; http://poweredby.revip.info)
Mozilla/5.0+(compatible; UptimeRobot/2.0; http://www.uptimerobot.com/)
Mozilla/5.0+(compatible;+PiplBot;++http://www.pipl.com/bot/)
Mozilla/5.0+(compatible;+PiplBot;+http://www.pipl.com/bot/)
Mozilla/5.0, Google-AdWords-Express
Mozilla/5.1 (compatible; +5Bot/1.0; +http://plus5files.net)
Mozilla/5.1 (compatible; DBot/7.5.6; +http://a14download.com)
Mozilla/5.1 (compatible; GLBot/1.0; +http://galaxydownloads.co)
Mozilla/5.1 (compatible; MonoBot/1.0; +http://mono.name)
Mozilla/5.1 (compatible; PWBot/1.0; +http://opendownload.pw)
Mozilla/5.5 (compatible; MSIE 5.5; Windows NT 5.1) Sitespider+ b432.1
MozillaTest/5.0 (compatible; YodaoBot/1.0; http://www.yodao.com/help/webmaster/spider/; )
Mozzila/5.0 (compatible; Sonic/1.0; http://www.yama.info.waseda.ac.jp/~crawler/info.html)
Mozzila/5.0 (compatible; YamanaLab-Robot/1.0; http://www.yama.info.waseda.ac.jp/~crawler/info.html)
MrakyPraceCzBot/1.0 (http://www.mrakyprace.cz/about-crawler)
MultiCrawler, http://sw.deri.org/2006/04/multicrawler/robots.html
My Nutch Spider/Nutch-1.6
My Spider 1.0,gzip(gfe),gzip(gfe)
MySpider/Nutch-2.2
NAVER Blog Rssbot
NETCRAFT
NFReader/1.4.1.0 (http://www.gaijin.at/)
NG-Search/0.86 (+http://www.ng-search.com)
NIF/1.1  (http://www.newsisfree.com/robot.php  users:)
NING/1.0
NINJA bot
NMG Spider/0.3 (szukanko.com)
NWSpider 0.9
NalezenCzBot/1.0 (http://www.nalezen.cz)
NalezenCzBot/1.0 (http://www.nalezen.cz/about-crawler)
NationalDirectory-WebSpider/1.3
Nekstbot - http://www.ipipan.waw.pl/nekst/nekstbot/
NerdyBot
NetLyzer FastProbe
NetLyzer FastProbe (See http://netlyzer.com/report/www.domain.com for info)
NetNewsWire/2.x (Mac OS X; http://ranchero.com/netnewswire/)
NetResearchServer(http://www.look.com)
NetResearchServer/2.5(loopimprovements.com/robot.html)
NetResearchServer/2.7(loopimprovements.com/robot.html)
NetResearchServer/2.8(loopimprovements.com/robot.html)
NetResearchServer/3.5(loopimprovements.com/robot.html)
NetResearchServer/4.0(loopimprovements.com/robot.html)
NetSurf/2.0 (RISC OS; armv5l)
NetSystemsResearch studies the availability of various services across the internet. Our website is netsystemsresearch.com
NetTrack Anonymous Web Statistics https://nettrack.info/support.php
NetTrack.info Crawler
NetWhatCrawler/0.06-dev (NetWhatCrawler from NetWhat.com; http://www.netwhat.com; support@netwhat.com)
Netchart Adv Crawler 1.0
Netcraft SSL Server Survey - contact info@netcraft.com
Neticle Crawler v1.0 ( https://neticle.com/bot/en/ )
NetpeakCheckerBot
Netvibes (crawler/bot; http://www.netvibes.com
Netvibes (crawler; http://www.netvibes.com)
Netvibes (http://www.netvibes.com)
Netvibes (http://www.netvibes.com/; 8 subscribers; feedID: 2244192)
NetzCheckBot/1.0 (security seal; private-krankenversicherung.de.netzcheck.com; bot@netzcheck.com)
NeumobBot/0.2.16 (+http://www.neumob.com/bot)
New-Sogou-Spider/1.0 (compatible; MSIE 5.5; Windows 98)
NewsBlur Favicon Fetcher - 7 subscribers - http://www.newsblur.com/site/1948420/analytics-piwik (Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_1) AppleWebKit/534.48.3 (KHTML, like Gecko) Version/5.1 Safari/534.48.3)
NewsBlur Feed Fetcher - 7 subscribers - http://www.newsblur.com/site/1948420/analytics-piwik (Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_1) AppleWebKit/534.48.3 (KHTML, like Gecko) Version/5.1 Safari/534.48.3)
NewsBlur Feed Finder (Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_1) AppleWebKit/534.48.3 (KHTML, like Gecko) Version/5.1 Safari/534.48.3)
NewsBlur Page Fetcher - 7 subscribers - http://www.newsblur.com/site/3966817/analytics-piwik (Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_1) AppleWebKit/534.48.3 (KHTML, like Gecko) Version/5.1 Safari/534.48.3)
NewsGator FetchLinks extension/0.2.0 (http://graemef.com)
NewsGator/2.0 Bot (http://www.newsgator.com)
NewsGator/2.5 (http://www.newsgator.com; Microsoft Windows NT 5.1.2600.0; .NET CLR 1.1.4322.2032)
NewsGatorOnline/2.0 (http://www.newsgator.com; 2 subscribers)
NewzCrawler/1.9 (compatible; MSIE 6.00; Newz Crawler 1.9; http://www.newzcrawler.com/ )
NextCloud-News/1.0
NextGenSearchBot 1 (for information visit http://www.zoominfo.com/About/misc/NextGenSearchBot.aspx)
NextGenSearchBot 69 (for information visit http://www.zoominfo.com/business/nextgensearchbot.html)
NihilScio - Educational search engine - +https://www.nihilscio.it/NihilScio.htm
NodejsSpider/1.0
Nokia6820/2.0 (4.83) Profile/MIDP-1.0 Configuration/CLDC-1.0 (compatible; Googlebot-Mobile/2.1; +http://www.google.com/bot.html)
Norbert the Spider(Burf.com)
Notifixious/LinkChecker (http://notifixio.us)
NuSearch Spider www.nusearch.com
Nuhk/2.4 ( http://www.neti.ee/cgi-bin/abi/Otsing/Nuhk/)
Nuhk/2.4 (+http://www.neti.ee/cgi-bin/abi/otsing.html)
Nusearch Spider (www.nusearch.com)
Nutch Checksem/Nutch-1.10
Nutch/2.2.1 (page scorer; http://integralads.com/site-indexing-policy/)
Nutch12/Nutch-1.2
NutchCVS/0.06-dev (Nutch running at UW; http://www.nutch.org/docs/en/bot.html; sycrawl@cs.washington.edu)
NutchCVS/0.7.1 (Nutch; http://lucene.apache.org/nutch/bot.html; nutch-agent@lucene.apache.org)
NutchOSUOSL/0.05-dev (Nutch; http://www.nutch.org/docs/en/bot.html; nutch-agent@lists.sourceforge.net)
NutchOrg/0.03-dev (Nutch; http://www.nutch.org/docs/bot.html; nutch-agent@lists.sourceforge.net)
NutchSpider/Nutch-1.4
Nuzzel
Nymesis/1.0 (http://nymesis.com)
Nymesis/2.0 (http://nymesis.com)
OMGCrawler 1.0
OOZBOT/0.20 ( -- ; http://www.setooz.com/oozbot.html ; agentname at setooz dot_com )
OOZBOT/0.20 ( Setooz výrazný ako say-th-uuz, znamená mosty.  ; http://www.setooz.com/oozbot.html ; agentname at setooz dot_com )
OOZBOT/0.20 ( http://www.setooz.com/oozbot.html ; agentname at setooz dot_com )
OSPScraper (+https://www.opensyllabusproject.org)
ObjectsSearch/0.2 (ObjectsSearch; http://www.ObjectsSearch.com/bot.html; support@thesoftwareobjects.com)
Ocarinabot
Ocelli/1.4 (http://www.globalspec.com/Ocelli)
Octopus 1.0.2
OdklBot/1.0 (klass@odnoklassniki.ru)
OhDear.app (+https://ohdear.app/docs/checks/uptime)
OmniExplorer_Bot/5.20 (+http://www.omni-explorer.com) WorldIndexer
OmniExplorer_Bot/5.91c (+http://www.omni-explorer.com) WorldIndexer
OmniExplorer_Bot/6.47 (+http://www.omni-explorer.com) WorldIndexer
OmniExplorer_Bot/6.60 (+http://www.omni-explorer.com) WorldIndexer
OmniExplorer_Bot/6.65a (+http://www.omni-explorer.com) WorldIndexer
OnPageBot (compatible; Googlebot 2.1; +https://bot.onpage.org/)
OnalyticaBot
Online Virus Scanner: http://tools.geek-tools.org
Open Web Analytics Bot 1.5.4
OpenLink Virtuoso RDF crawler
OpenWebSpider v0.1.2.B (http://www.openwebspider.org/)
OpenWebSpider v0.1.4 (http://www.openwebspider.org/)
OpenWebSpider/0.6 (http://www.openwebspider.org)
Opendi Screenshot Bot bot@opendi.com
Opera/8.01 (J2ME/MIDP; MXit WebBot/1.8.3.119) Opera Mini/3.1
OpmlViewer network fetcher version 1.0.0.0
OrangeCrawler/Nutch-1.0 (ldorange.crawler@orange-ftgroup.com)
Orbiter ( http://www.dailyorbit.com/bot.htm)
Orbiter (+http://www.dailyorbit.com/bot.htm)
Orbiter/1.0 (http://dailyorbit.com/)
Orbiter/1.2 (http://dailyorbit.com/)
Orbiter/1.3 (http://dailyorbit.com/)
OrgProbe/0.9.4 (+http://www.blocked.org.uk)
Orgbybot/OrgbyBot v1.2 (Spidering the net for Orgby; http://www.orgby.com/  ; Orgby.com Search Engine)
Orgbybot/OrgbyBot v1.2 (Spidering the net for Orgby; http://www.orgby.com/ ; Orgby.com Search Engine)
Orgbybot/OrgbyBot v1.3 (Spider; http://orgby.com/bot/  ; Orgby.com Search Engine)
OutclicksBot/2 +https://www.outclicks.net/agent/PryJzTl8POCRHfvEUlRN5FKtZoWDQOBEvFJ2wh6KH5J
OutclicksBot/2 +https://www.outclicks.net/agent/VjzDygCuk4ubNmg40ZMbFqT0sIh7UfOKk8s8ZMiupUR
OutclicksBot/2 +https://www.outclicks.net/agent/gIYbZ38dfAuhZkrFVl7sJBFOUhOVct6J1SvxgmBZgCe
OutclicksBot/2 +https://www.outclicks.net/agent/p2i4sNUh7eylJF1S6SGgRs5mP40ExlYvsr9GBxVQG6h
Overture-WebCrawler/3.8/Fresh (atw-crawler at fast dot no; http://fast.no/support/crawler.asp)
PArchiveCrawler/1.0
PDFBot (crawler@pdfind.com)
PEAR HTTPRequest class ( http://pear.php.net/ )
PHP-Curl-Class/4.13.0 (+https://github.com/php-curl-class/php-curl-class) PHP/7.2.24 curl/7.61.1
PHP-Curl-Class/4.13.0 (+https://github.com/php-curl-class/php-curl-class) PHP/7.3.19 curl/7.66.0
PHP-Curl-Class/4.13.0 (+https://github.com/php-curl-class/php-curl-class) PHP/7.3.23 curl/7.66.0
PHP-Curl-Class/4.13.0 (+https://github.com/php-curl-class/php-curl-class) PHP/7.4.10 curl/7.69.1
PHP-Curl-Class/4.13.0 (+https://github.com/php-curl-class/php-curl-class) PHP/7.4.11 curl/7.69.1
PHP-Curl-Class/4.13.0 (+https://github.com/php-curl-class/php-curl-class) PHP/7.4.7 curl/7.69.1
PHP-Curl-Class/4.13.0 (+https://github.com/php-curl-class/php-curl-class) PHP/7.4.9 curl/7.69.1
PHP/5.2.13
PHP/5.5
PHP/{5|6|7}.{3|2}.{1|2|3|4|5|6|7|8|9|0}{1|2|3|4|5|6|7|8|9|0}
PHPCrawl
PINGOMETER_BOT_(HTTPS://PINGOMETER.COM)
PKU Student Spider
Page Analyzer v4.0 ( http://www.ranks.nl/ )
PageBitesHyperBot/600 (http://www.pagebites.com/)
PageFreezer (pagefreezer crawler; http://pagefreezer.com/pagefreezer-crawler/; info@pagefreezer.com)
PagePeeker.com
PagePeeker.com (info: http://pagepeeker.com/robots)
PageThing http://pagething.com
PageThing.com
PagesInventory (robot +http://www.pagesinventory.com)
PagesInventory (robot http://www.pagesinvenotry.com)
PagesInventory (robot http://www.pagesinventory.com)
PagesInventory.com (robot http://www.pagesinvenotry.com)
Pandalytics/1.0 (https://domainsbot.com/pandalytics/)
Pattern/2.6 +http://www.clips.ua.ac.be/pattern
Patwebbot (http://www.herz-power.de/technik.html)
PayPal IPN ( https://www.paypal.com/ipn )
Pcore-HTTP/v0.23.20
Pcore-HTTP/v0.40.3
Pcore-HTTP/v0.44.0
Peeplo Screenshot Bot/0.20 ( abuse at peeplo dot_com )
PeerFactor Crawler
PercolateCrawler/3.1.30 (ops@percolate.com)
PercolateCrawler/4 (ops@percolate.com)
Perlu - HTML Analyzer and Relationship Builder 1.1
Pete-Spider/1.1
Phantom.js bot
PhoenixWebBot Beta
PicSpider/1.1 (spider@217-20-118-26.internetserviceteam.com; http://www.bildkiste.de)
PingSpot/1.0 (http://www.pingspot.org/)
Pingability.com Monitoring Service
Pingdom GIGRIB (http://www.pingdom.com)
Pingdom.com_bot_version_1.4_(http://www.pingdom.com)
Pingdom.com_bot_version_1.4_(http://www.pingdom.com/)
Pinterest/0.1 (+https://www.pinterest.com/bot.html)
Pinterest/0.1 +http://pinterest.com/
Pinterest/0.2 (+http://www.pinterest.com/)
Pinterest/0.2 (+http://www.pinterest.com/bot.html)
Pinterest/0.2 (+https://www.pinterest.com/bot.html)
PiplBot (+http://www.pipl.com/bot/)
Pixray-Seeker/1.1 (Pixray-Seeker; crawler@pixray.com)
Pixray-Seeker/1.1 (Pixray-Seeker; http://www.pixray.com/pixraybot; crawler@pixray.com)
Pixray-Seeker/2.0 (Pixray-Seeker; +http://www.pixray.com/pixraybot; +crawler@pixray.com)
Pixray-Seeker/2.0 (Pixray-Seeker; http://www.pixray.com/pixraybot; crawler@pixray.com)
Pixray-Seeker/2.0 (http://www.pixray.com/pixraybot; crawler@pixray.com)
Plagger/0.x.xx (http://plagger.org/)
PlantyNet_WebRobot_V1.9 babo@plantynet.com
Pleroma 2.1.50-495-gf9ece1a7-develop+dev; https://search.fedi.app <pleroma@search.fedi.app>; Bot
Ploetz + Zeller (http://www.ploetz-zeller.de) Link Validator v1.0 (support@p-und-z.de) for ARIS Business Architect
PluckItCrawler/1.0 (compatible; Mozilla 4.0; MSIE 5.5; http://www.pluck.com;)
PocketParser/2.0 (+https://getpocket.com/pocketparser_ua)
PodcastAddict/v5 (+https://podcastaddict.com/; Android podcast app)
Pompos/1.2 http://pompos.iliad.fr
Pompos/1.3 http://dir.com/pompos.html
Porkbun/Mustache (Website Analysis; http://porkbun.com; tech@porkbun.com)
Port Monitor check service 1.0 | http://www.port-monitor.com
PortalBSpider/2.0 (spider@portalb.com)
PostPost/1.0 ( http://postpo.st/crawlers)
PostPost/1.0 (+http://postpo.st/crawlers)
PostPost/1.0 (+http://postpost.com/crawlers)
PostRank/2.0 (postrank.com)
PostRank/2.0 (postrank.com; 1 subscribers)
PostmanRuntime/7.1.1
Prebid.js Scraper
PritTorrent/0.1
PritTorrent/1.0
PrivateSearch/0.1.0 (Polite Nutch Crawler; grierforensics.com)
ProjectWF-java-test-crawler
ProloCrawler/1.0 (http://www.prolo.com)
ProoXiBot
Protopage/3.0 (http://www.protopage.com)
PsSpider/0.7 (spider@bildkiste.de)
Pu_iN Crawler (+http://semanticjuice.com/)
Pulp/1.5.2 (iPad; http://www.acrylicapps.com/pulp/)
Pulsepoint XT3 web scraper
Punk Spider/PunkSPIDER-v0.1
Putin (+http://semanticjuice.com/)
PycURL/7.22.0
Python-urllib/1.17
Python-urllib/2.5
Python-urllib/2.6
Python-urllib/2.7
Python-urllib/2.7 (+http://aranea.juls.savba.sk/aranea_about.html)
Python-urllib/2.7 (+http://ella.juls.savba.sk/aranea_about)
Python-urllib/2.7 (+http://sketch.juls.savba.sk/aranea_about)
Python-urllib/3.0
Python-urllib/3.1
Python-urllib/3.2
Python-urllib/3.3
Python-urllib/3.4
Python-urllib/3.5
Python-urllib/3.6
Python-urllib/3.7
Python/3.5 aiohttp/0.21.6
Python/3.7 aiohttp/3.6.2a2
Python/3.8 aiohttp/3.7.2
Python/3.9 aiohttp/3.7.3
Qirina Hurdler v. 1.05 10.11.01 (+http://www.qirina.com/hurdler.html)
QuerySeekerSpider ( http://queryseeker.com/bot.html )
Quick-Crawler (+https://www.scrapinghub.com/)
QuickFinder Crawler
QuickiWiki/1.0 (http://www.quickiwiki.com; admin@quickiwiki.com) BasedOnNodeJs/0.10.23
Quora Link Preview/1.0 (http://www.quora.com)
Quora-Bot/1.0 (http://www.quora.com)
Qwantify/1.0
R6_CommentReader(www.radian6.com/crawler)
R6_FeedFetcher(www.radian6.com/crawler)
R6_FeedFetcher_(www.radian6.com/crawler)
RADaR-Bot/Nutch-1.3 (http://radar-bot.com/)
RAMPyBot - www.giveRAMP.com/0.1 (RAMPyBot - www.giveRAMP.com; http://www.giveramp.com/bot.html; support@giveRAMP.com)
RB2B-bot v0.1 (Using Fast Enterprise Crawler 6 search@reedbusiness.com)
RED/1 (http://redbot.org/)
RED/1 (https://redbot.org/)
REL Link Checker Lite 1.0
ROI Hunter; https://api-dev.roihunter.com
ROR Sitemap Generator (http://www.rorweb.com)
RSS-SPIDER (RSS Feed Seeker http://www.MyNewFavoriteThing.com/fsb.php)
RSSIncludeBot/1.0 (http://www.rssinclude.com/spider)
RSSMicro.com RSS/Atom Feed Robot
RSSingBot (http://www.rssing.com)
RSpider/1.0
Rainmeter WebParser plugin
RamblerMail/6.0 (incompatible; ImageProxy/6.0)
RankFlex.com Webspider
RankurBot/3.3 (+http://rankur.com)
RankurBot/Rankur2.1 (http://rankur.com; info at rankur dot com)
Raven Link Checker
RavenCrawler
Re-re Studio (+http://re-re.ru/)
ReadPath_Spider
Readability/1900e6 - http://readability.com/about/
Readability/6a54d4 - http://readability.com/about/
Readability/740ec9 - http://readability.com/about/
Reaper/2.07 ( http://www.sitesearch.ca/reaper)
RebelMouse/0.1 Mozilla/5.0 (compatible; http://rebelmouse.com) Gecko/20100101 Firefox/7.0.1
RecurPost/1.0 https://recurpost.com (KHTML, like Gecko)
RedBot/redbot-1.0 (Rediff.com Crawler; redbot at rediff dot com)
RedCarpet/1.0 (http://www.redcarpet-inc.com/robots.html)
RedesScrapy/0.24.1 (+http://g2pi.tsc.uc3m.es/es)
RelateIQ Crawler www.relateiq.com
Research spider - ak1835@albany.edu
Riddler (http://riddler.io/about)
Riddler (http://riddler.io/about.html)
RobotsChecker/0.6 (+http://www.blocked.org.uk)
Robots_Tester_http_www.searchenginepromotionhelp.com
Robozilla/1.0
RocketMonitorBot/1.0 ( https://www.radiomast.io)
Ronzoobot/1.2 (http://www.ronzoo.com/about.php)
Ronzoobot/1.3 (http://www.ronzoo.com/about.php)
Ronzoobot/1.5 (http://www.ronzoo.com/about/)
Ronzoobot/1.6 (http://www.ronzoo.com/about/)
RssReader/1.0.xx.x (http://www.rssreader.com) Microsoft Windows NT 5.1.2600.0
Ruby, Twurly v1.1 (http://twurly.org)
RufusBot (Rufus Web Miner; http://64.124.122.252/feedback.html)
Ruky-Roboter (Version: 1.06, powered by www.ruky.de +http://www.ruky.de/bot.html)
RyteBot/1.0.0 (+https://bot.ryte.com/)
RyzeCrawler/1.1.0 (+http://www.ryze.nl/crawler/)
RyzeCrawler/1.1.1 ( http://www.domain2day.nl/crawler/)
RyzeCrawler/1.1.1 (+http://www.domain2day.nl/crawler/)
SAE/fetchurl-22wy0njxnm WordPress/3.2.1; http://1.webfront.sinaapp.com
SAMSUNG-SGH-E250/1.0 Profile/MIDP-2.0 Configuration/CLDC-1.1 UP.Browser/6.2.3.3.c.1.101 (GUI) MMP/2.0 (compatible; Googlebot-Mobile/2.1; +http://www.google.com/bot.html)
SBIder/0.8-dev (SBIder; http://www.sitesell.com/sbider.html; http://support.sitesell.com/contact-support.html)
SBIder/Nutch-1.0-dev (http://www.sitesell.com/sbider.html)
SBL-BOT (http://sbl.net)
SCFCrawler/Nutch-1.8 (Image Crawler for StolenCameraFinder.com; http://www.stolencamerafinder.com/; crawler@stolencamerafinder.com)
SCrawlTest/CR1 (CWD; http://sproose.com; crawl@sproose.com)
SEMrushBot
SEO Consulting; Redirect Checker Tool V.02; IP:
SEOCentro Page Keyword Analyzer v1.2
SEOENGBot/1.2 (+http://learn.seoeng.com/seoengbot.htm)
SEOENGBot/1.2 (+http://learn.seoengine.com/seoengbot.htm)
SEOENGBot/1.2 (+http://www.seoengine.com/seoengbot.htm)
SEOENGWorldBot/1.0 (+http://www.seoengine.com/seoengbot.htm)
SEOlizer/1.1 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.13) Gecko/20080311 Firefox/2.0.0.13 (+https://www.seolizer.de/bot.html)
SEOstats 2.0.9 https://github.com/eyecatchup/SEOstats
SEOstats 2.1.0 https://github.com/eyecatchup/SEOstats
SETOOZBOT/5.0 ( compatible; SETOOZBOT/0.30 ; http://www.setooz.com/bot.html )
SETOOZBOT/5.0 ( http://www.setooz.com/bot.html )
SMTBot (similartech.com/smtbot)
SMXCrawler/1.0 (www.socialmetrix.com)
SNK Screenshot Bot/0.20 ( abuse at savenkeep dot_com )
SPDYCheck SPDY Protocol Tester, see http://spdycheck.org/about.html
SPIP-3.0.20 (http://www.spip.net)
SSL Labs (https://www.ssllabs.com/about/assessment.html)
SWAT Crawler. AGH University project. In case of problem contact: opal@tempus.metal.agh.edu.pl. Thanks.
SafeAds.xyz bot
SafeDNS search bot/Nutch-1.9 (https://www.safedns.com/searchbot; support [at] safedns [dot] com)
SafeDNSBot (https://www.safedns.com/searchbot)
SafeSearch microdata crawler (https://safesearch.avira.com, safesearch-abuse@avira.com)
SaladSpoon/ShopSalad 1.0 (Search Engine crawler for ShopSalad.com; http://shopsalad.com/en/partners.html; crawler AT shopsalad.com)
SalesIntelligent (+https://www.salesintelligent.com)
SalesIntelligent/v1.1 (https://www.salesintelligent.com)
SandCrawler - Compatibility Testing
Sangfor Spider/0.7
SanszBot/1.7(http://www.sansz.org/sanszbot, spider@sansz.org) (spider@sansz.org)
SapphireWebCrawler/1.0 (Sapphire Web Crawler using Nutch; http://boston.lti.cs.cmu.edu/crawler/; mhoy@cs.cmu.edu)
SauceNAO/1.0 (+http://saucenao.com/)
ScSpider/0.2
Scopia crawler 1.0 (+http://www.scopia.co)
Scrapy/0.16.5 (+http://scrapy.org)
Scrapy/0.18.4 (+http://scrapy.org)
Scrapy/0.20.1 (+http://scrapy.org)
Scrapy/0.22.2 (+http://scrapy.org)
Scrapy/0.24.0 (+http://scrapy.org)
Scrapy/0.24.4 (+http://scrapy.org)
Scrapy/0.24.5 (+http://scrapy.org)
Scrapy/0.24.6 (+http://scrapy.org)
Scrapy/0.25.1 (+http://scrapy.org)
Scrapy/1.0.1 (+http://scrapy.org)
Scrapy/1.0.3 (+http://scrapy.org)
Scrapy/1.0.3.post6+g2d688cd (+http://scrapy.org)
Scrapy/1.0.4 (+http://scrapy.org)
Scrapy/1.0.5 (+http://scrapy.org)
Scrapy/1.0.5.post4+g4b324a8 (+http://scrapy.org)
Scrapy/1.1.0 (+http://scrapy.org)
Scrapy/1.1.1 (+http://scrapy.org)
Scrapy/1.1.2 (+http://scrapy.org)
Scrapy/1.2.1 (+http://scrapy.org)
Scrapy/1.3.0 (+http://scrapy.org)
Screaming Frog SEO Spider/2,01
Screaming Frog SEO Spider/2.22
Screaming Frog SEO Spider/5.1
ScreenerBot Crawler Beta 2.0 (+http://www.ScreenerBot.com)
Scrubby/2.1 (http://www.scrubtheweb.com/)
Scrubby/2.2 (http://www.scrubtheweb.com/)
Scrubby/3.0 (+http://www.scrubtheweb.com/help/technology.html)
Search37/1.0 (http://www.search37.com; info@search37.com)
Search37/1.2 (http://www.search37.com; info@search37.com)
SearchAtlas.com SEO Crawler
SearchSight/2.0 (http://SearchSight.com/)
SearchSpider.com/1.1
Searchie Bot/1.0 (a Storm-based crawler; https://www.searchie.org; admin@searchie.org)
Searchie/1.0 (a Storm-based crawler; https://www.searchie.org; admin@searchie.org)
SeekOn Spider 1.9(+http://www.seekon.com/spider.html)
Seekbot/1.0 (http://www.seekbot.net/bot.html) RobotsTxtFetcher/1.2
Seeker v.1 http://LookSeek.com/seeker/
SemanticScholarBot/1.0 (+http://s2.allenai.org/bot.html)
Semantifire1/0.20 ( -- ; http://www.setooz.com/oozbot.html ; agentname at setooz dot_com )
Semantifire1/0.20 ( http://www.setooz.com/oozbot.html ; agentname at setooz dot_com )
SemrushBot/0.9
SemrushBot/0.91
SemrushBot/0.92
SemrushBot/Nutch-1.5-SNAPSHOT
Sendsay.Ru/1.0; https://Sendsay.Ru/; ask@sendsay.ru
Sens.ai http://sovereign.ai/contact
SensikaBot/x.33 (+http://sensika.com)
Sensis Web Crawler (search_comments\at\sensis\dot\com\dot\au)
SentiBot www.sentibot.eu (compatible with Googlebot)
SeoCheck (FischerNetzDesign Seo Checker, info@fischernetzdesign.de)
SeoCheckBot (FischerNetzDesign Seo Checker, info@fischernetzdesign.de)
SeoCheckBot (Seo-Check, http://www.kfsw.de/bot.html)
SeoCheckBot (Seomira Seo Check, http://www.kfsw.de/bot.html)
Seobility
Seobility (SEO-Check; http://bit.ly/1dJuuzs)
SeobilityBot (SEO Tool; https://www.seobility.net/sites/bot.html)
SeobilityBot (SEO-Check; http://bit.ly/1dJuuzs)
Seologies/0.1 Bot
SeopultContentAnalyzer/1.0
SerendeputyBot/0.8.6 (http://serendeputy.com/about/serendeputy-bot)
Server Density Service Monitoring v2
ServiceUptime.robot
SetCronJob/1.0 (+http://www.setcronjob.com/)
SetLinks bot 1.0
Setooz/Nutch-1.0 (http://www.setooz.com)
Seznam-Zbozi-robot/3.0
Seznam-Zbozi-robot/3.2.1
Seznam-Zbozi-robot/3.2.2
Seznam-Zbozi-robot/3.3
SeznamBot/2.0 (+http://fulltext.sblog.cz/robot/)
SeznamBot/2.0 (+http://fulltext.seznam.cz/)
SeznamBot/2.0-Test (+http://fulltext.sblog.cz/robot/)
SeznamBot/3.0 (+http://fulltext.sblog.cz/)
SeznamBot/3.0 (HaF+http://fulltext.sblog.cz/)
SeznamBot/3.0-alpha (+http://fulltext.sblog.cz/)
SeznamBot/3.0-beta (+http://fulltext.sblog.cz/)
SeznamBot/3.0-beta (+http://fulltext.sblog.cz/), I
SeznamBot/3.0-test (+http://fulltext.sblog.cz/)
SeznamBot/3.0-test (+http://fulltext.sblog.cz/), I
SheenBot/SheenBot-1.0.4 (Sheen web crawler)
Shim-Crawler
ShopWiki/1.0 ( +http://www.shopwiki.com/wiki/Help:Bot)
Short URL Checker ( http://www.kurld.com/ )
ShowyouBot (http://showyou.com/crawler)
Sideqik +http://www.sideqik.com
SilverReader/1.0; http://silverreader.com
SimpleChecker
SimpleCrawler/0.1
SimplePie/1.2.1-dev (Feed Parser; http://simplepie.org; Allow like Gecko) Build/20130514092120
SimplePie/1.3-dev (Feed Parser; http://simplepie.org; Allow like Gecko)
SimplePie/1.3.1 (Feed Parser; http://simplepie.org; Allow like Gecko) Build/20121030175911
SimplePie/1.3.1 (Feed Parser; http://simplepie.org; Allow like Gecko) Build/20160226060242
Simpy/1.1 (Simpy; http://www.simpy.com/?ref=bot; feedback at simpy dot com)
Site24x7
Site24x7 Agent
Site24x7 Tools
SiteAnalyzerBot
SiteBar/3.3.5 (Bookmark Server; http://sitebar.org/)
SiteBar/3.3.8 (Bookmark Server; http://sitebar.org/)
SiteCheck - http://downforeveryoneorjustme.com
SiteCrawler,gzip(gfe),gzip(gfe)
SiteGuardian/2.0 (Internet Monitoring)
SiteSucker for macOS/2.10.5
SiteSucker for macOS/2.10.6
SiteSucker/2.4.6
SiteUptime.com
Sitedomain-Bot(Sitedomain-Bot 1.0, http://www.sitedomain.de/sitedomain-bot/)
SitemapParser-VIPnytt/1.1 ( https://github.com/VIPnytt/SitemapParser/blob/master/README.md)
SklikBot/2.0 (sklik@firma.seznam.cz;+http://napoveda.sklik.cz/)
Slack-ImgProxy (+https://api.slack.com/robots)
Slack-ImgProxy 0.59 (+https://api.slack.com/robots)
Slack-ImgProxy 0.66 (+https://api.slack.com/robots)
Slack-ImgProxy 1.106 (+https://api.slack.com/robots)
Slack-ImgProxy 1.138 (+https://api.slack.com/robots)
Slack-ImgProxy 149 (+https://api.slack.com/robots)
Slackbot (+https://api.slack.com/robots)
Slackbot 1.0 (+https://api.slack.com/robots)
Slackbot-LinkExpanding (+https://api.slack.com/robots)
Slackbot-LinkExpanding 1.0 (+https://api.slack.com/robots)
Slackbot-LinkExpanding+(+https://api.slack.com/robots)
Slinky v1.0 +http://dentedreality.com.au/projects/slinky/
SlugBug Spider/0.1 beta (SlugBug.com search engine; http://www.slugbug.com)
Slurp/2.0
SmabblerBot/1.0 (http://www.smabbler.com/)
SmartAndSimpleWebCrawler/1.3 (https://crawler.dev.java.net),gzip(gfe),gzip(gfe)
Smartsite HTTPClient - Mozilla/4.0 (compatible; MSIE 6.0)
SmerBot/0.1 (smerity@smerity.com) Language Model Dataset Fetcher
Snapbot/1.0 (Snap Shots, +http://www.snap.com)
Snappy/1.1 ( http://www.urltrends.com/ )
Snappy/2.0 ( http://www.urltrends.com/ )
Snarfer/0.x.x (http://www.snarfware.com/)
SniffRSS/0.5beta (+http://www.blogator.com/)
Social Media Crawler using your Home URL on Twitter,Facebook,Myspace,Linkedin by ProfileCapture - contact profilecapture@gmail.com to report any problems with my crawling. http://profilecapture.com
SocialRankIOBot; http://socialrank.io/about
SocialSpider-Finder/0.2
Sogou Mobile Spider1.0 (http://wap.sogou.com)
Sogou News Spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm#07)
Sogou Orion spider/4.0( http://www.sogou.com/docs/help/webmasters.htm#07)
Sogou Pic Spider/3.0(+http://www.sogou.com/docs/help/webmasters.htm
Sogou Pic Spider/3.0(+http://www.sogou.com/docs/help/webmasters.htm#07)
Sogou Push Spider/3.0( http://www.sogou.com/docs/help/webmasters.htm#07)
Sogou Web Spider
Sogou develop spider/4.0( http://www.sogou.com/docs/help/webmasters.htm#07)
Sogou head spider/3.0( http://www.sogou.com/docs/help/webmasters.htm#07)
Sogou inst spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm
Sogou web spider/3.0( http://www.sogou.com/docs/help/webmasters.htm#07)
Sogou web spider/4.0
Sogou web spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm
Sogou web spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm#07)
Sogou web spider/4.025251
Sogou web spider/4.02525A
Sogou web spider/4.05252A
Sogou-Test-Spider/4.0 (compatible; MSIE 5.5; Windows 98)
SolomonoBot/1.04 (http://www.solomono.ru)
Sosoblogspider+(+http://help.soso.com/soso-blog-spider.htm)
Sosoimagespider ( http://help.soso.com/soso-image-spider.htm)
Sosospider+(+help.soso.com/webspider.htm)
Sosospider+(+http://help.soso.com/webspider.htm)
Speedy Spider (Entireweb; Beta/1.2; http://www.entireweb.com/about/search_tech/speedyspider/)
Speedy Spider (Entireweb; Beta/1.3; http://www.entireweb.com/about/search_tech/speedyspider/)
Speedy Spider (Submit your site at http://www.entireweb.com/free_submission/)
Speedy Spider (http://www.entireweb.com)
Speedy Spider (http://www.entireweb.com/about/search_tech/speedy_spider/)
SpeedySpider - http://www.entireweb.com
Spider/Nutch-2.3-SNAPSHOT (Webcrawler)
SpiderKU/0.9
SpiderMan 3.0.1-2-11-111 (CP/M;8-bit)
Spider_Monkey/7.06 (SpiderMonkey.ca info at http://SpiderMonkey.ca /sm.shtml)
SpokeSpider/1.0 (http://support.spoke.com/webspider/) Mozilla/5.0 (not really)
Spotify/1.0
Sprinklr 2.0
Statically-Images/2.0
Statically-Staticzap/2.0
StatoolsBot (+http://www.statools.com/bot.html)
StatusCake Error Detector Crawler V0.01
Steeler/3.3 (http://www.tkl.iis.u-tokyo.ac.jp/~crawler/)
StormCrawler/1.0 (a Storm-based crawler; https://github.com/DigitalPebble/storm-crawler; stormcrawler@digitalpebble.com)
Stripe/1.0 (+https://stripe.com/docs/webhooks)
Stroke.cz (http://stroke.cz)
Sufog/Nutch-2.2.1 (www.sufog.com; www.sufog.com)
Summify (Summify/1.0.1; +http://summify.com)
Super Monitoring
SuperPagesUrlVerifyBot/1.0
Superarama.com - BOT/v.0.1
Superarama.com-Tarama-Botu-v.0.1
Superarama.com-Tarama-Botu-v.01
Superfeedr bot/2.0 http://superfeedr.com - Make your feeds realtime: get in touch
Superfeedr bot/2.0 http://superfeedr.com - Make your feeds realtime: get in touch - feed-id:1162088860
Superfeedr bot/2.0 http://superfeedr.com - Make your feeds realtime: get in touch!
SurcentroBot
SurveyBot/2.3 (Whois Source)
Swiftbot
SygolBot http://www.sygol.com
Symfony Spider (http://symfony.com/spider)
SynHttpClient-Built/5759
SynHttpClient-Built/5825
SynooBot/0.7.1 (SynooBot; http://www.synoo.de/bot.html; webmaster@synoo.com)
Synthesio Crawler release MonaLisa (contact at synthesio dot fr)
Szukacz/1.5 (robot; www.szukacz.pl/jakdzialarobot.html; info@szukacz.pl)
T3census-Crawler/1.0
TA SEO Crawler v0.1 URLData.pm (smcquaker@tripadvisor.com)
TAMU_CS_IRL_CRAWLER/1.0
TBI-HttpOpenPlugin/0.1.0 (+https://leakix.net/)
TBI-WebScanner/0.0.1 (+https://leakix.net/)
TECHIAITH_CY (+https://techiaith.cymru)
TECOMAC-Crawler/0.3
TLS tester from https://testssl.sh/dev/
TLSProbe/1.0 (+https://scan.trustnet.venafi.com/)
TTop-Crawler,gzip(gfe),gzip(gfe)
TailsweepBlogCrawler/Tailsweep-2.9-SNAPSHOT (http://www.tailsweep.com/; bot at [tailsweep] dot com)
TangibleeBot
TangibleeBot/1.0.0.0 (http://tangiblee.com/bot)
Taproot (Guzzle) http://taprootproject.com
Tarantula/BETA-0.5 (Tarantula de la Enlle, el mejor buscador en la red; http://www.enlle.com; tarantula@enlle.com)
Tarmot Gezgin/1.0 (compatible; TarmotGezgin/1.1; +http://www.tarmot.com/gezgin)
Tasapspider/0.9
TayaCrawler (Beta; v0.1; tayadev@nexdegree.com)
TeamSpeak3-ImageFetcher-1.0
Technoratibot/7.0
Technoratibot/8.0
TelegramBot (like TwitterBot)
TelemetrySpider2/0.1 linux
Teragram/SAS Crawler
Testomatobot/1.0 (Linux x86_64; +http://www.testomato.com/testomatobot) minicrawler/3.0.0
Testomatobot/1.0 (Linux x86_64; +http://www.testomato.com/testomatobot) minicrawler/3.0.1
Testomatobot/1.0 (Linux x86_64; +http://www.testomato.com/testomatobot) minicrawler/3.0~beta
Testomatobot/1.0 (Linux x86_64; +http://www.testomato.com/testomatobot) minicrawler/4.0.0
Testomatobot/1.0 (Linux x86_64; +http://www.testomato.com/testomatobot) minicrawler/4.0.0~beta10
Testomatobot/1.0 (Linux x86_64; +http://www.testomato.com/testomatobot) minicrawler/4.0.0~beta11
Testomatobot/1.0 (Linux x86_64; +http://www.testomato.com/testomatobot) minicrawler/4.0.0~beta12
Testomatobot/1.0 (Linux x86_64; +http://www.testomato.com/testomatobot) minicrawler/4.0.0~beta6
Testomatobot/1.0 (Linux x86_64; +http://www.testomato.com/testomatobot) minicrawler/4.0.0~beta7
Testomatobot/1.0 (Linux x86_64; +http://www.testomato.com/testomatobot) minicrawler/4.0.0~beta8
Testomatobot/1.0 (Linux x86_64; +http://www.testomato.com/testomatobot) minicrawler/4.0.0~beta9
Testomatobot/1.0 (Linux x86_64; +http://www.testomato.com/testomatobot) minicrawler/4.1.0
Testomatobot/1.0 (Linux x86_64; +http://www.testomato.com/testomatobot) minicrawler/5.0.1
Testomatobot/1.0 (Linux x86_64; +http://www.testomato.com/testomatobot) minicrawler/5.0.2
Testomatobot/1.0 (Linux x86_64; +http://www.testomato.com/testomatobot) minicrawler/5.0.3
Testomatobot/1.0 (Linux x86_64; +http://www.testomato.com/testomatobot) minicrawler/5.0.4
TextRazor Downloader (https://www.textrazor.com)
The Expert HTML Source Viewer (http://www.expert-html.com)
The Knowledge AI
The Knowledge AI the.knowledge.ai@gmail.com
The Lemur Web Crawler/Nutch-1.3 (Lemur Web Crawler; http://boston.lti.cs.cmu.edu/crawler_12/; admin@lemurproject.org)
TheSuBot/0.1 (www.thesubot.de)
TheUsefulbot_2.3.62 (bot@theusefulnet.com)
Theme Spider ( http://www.themespider.com/spider.html)
Thesis_php_crawler
Thinklab (thinklab.com)
ThumbShots-Bot (+http://thumbshots.in/bot.html)
ThumbSniper (http://thumbsniper.com)
Thumbnail.CZ robot 1.1 (http://thumbnail.cz/why-no-robots-txt.html)
TinEye
TinEye-bot/0.02 (see http://www.tineye.com/crawler.html)
TinEye-bot/0.51 (see http://www.tineye.com/crawler.html)
TinEye-bot/0.61 (see http://www.tineye.com/crawler.html)
TinEye/1.0; +http://www.tineye.com/
TinEye/1.1 (http://tineye.com/crawler.html)
Tiny Tiny RSS/1.10 (http://tt-rss.org/)
Tiny Tiny RSS/1.11.4c63934 (http://tt-rss.org/)
Tiny Tiny RSS/1.15.3 (http://tt-rss.org/)
Tiny Tiny RSS/17.12 (a2d1fa5) (http://tt-rss.org/)
Tiny Tiny RSS/19.2 (b68db2d) (http://tt-rss.org/)
Tiny Tiny RSS/19.8 (http://tt-rss.org/)
TipTop http://feeltiptop.com
Toms Spider 0.3
Tools4noobs.com/1.0 Spider
Top10Ranking Spider/3.1 ( http://www.top10Ranking.nl/, Top10ranking.nl heeft op een aantal woorden uw posities in Google gecheckt)
Topicbot/1.0 (Mozilla;I;+http://92.42.190.57/)
TouTrix crawler/1.0
Touche (+http://www.touche.com.ve)
Toweyabot: toweya.com
Traackr.com
Trackuity (+http://trackuity.com)
TrapitAgent/0.1 (feed processor; +http://trapit.com/about)
Trends Crawler, Real time trends bot (info@trendscrawler.com)
Trove (help@trove.com, http://www.trove.com)
TulipChain/5.xx (http://ostermiller.org/tulipchain/) Java/1.x.1_0x (http://apple.com/) Mac_OS_X/10.2.8
Turnitin (https://bit.ly/2UvnfoQ)
TurnitinBot (https://turnitin.com/robot/crawlerinfo.html)
TurnitinBot/2.1 (http://www.turnitin.com/robot/crawlerinfo.html)
TurnitinBot/3.0 (http://www.turnitin.com/robot/crawlerinfo.html)
Tutorial Crawler 1.4
TweetedTimes Bot/1.0 (Mozilla/5.0 Compatible, +http://tweetedtimes.com)
TwengaBot
TwengaBot-2.0
TwengaBot-2.0 Champigny (+http://www.twenga.com/bot.html)
TwengaBot-Discover (http://www.twenga.fr/bot-discover.html)
TwengaBot/1.1 (+http://www.twenga.com/bot.html)
Twikle/1.0 , http://twikle.com , contact@twikle.com
TwitSpider
Twitmunin Crawler http://www.twitmunin.com
Twitterbot/0.1
Twitterbot/1.0
Typhoeus - https://github.com/typhoeus/typhoeus
UCMore Crawler App
UOLCrawler (soscrawler@uol.com.br)
URLChecker
USyd-NLP-Spider (http://www.it.usyd.edu.au/~vinci/bot.html)
UT-Dorkbot/1.0
UXCrawlerBot
UnisterBot (Mozilla/5.0 compatible; crawler@unister-gmbh.de)
UnityPlayer/4.3.2f1 (http://unity3d.com)
UniversalFeedParser/3.3 +http://feedparser.org/
UniversalFeedParser/5.1.3 +https://code.google.com/p/feedparser/
UniversalFeedParser/5.2.1 +https://code.google.com/p/feedparser/
Unshorten.It!/1.0 (https://unshorten.it/)
UnwindFetchor/1.0 (+http://www.gnip.com/)
Updownerbot (+http://www.updowner.com/bot)
Upflow/1.0
Uptime.com_(http://uptime.com/)
UptimeDog Robot (www.uptimedog.com)
Uptimebot.org - Free website monitoring
UrlTrends Ranking Updater/2.0 ( http://www.urltrends.com/ )
Urlstat (http://bit.ly/1dJuuzs)
VORTEX/1.2 ( http://marty.anstey.ca/robots/vortex/)
VSB-TUO/1.0 (+http://www.vsb.cz)
VSE/1.0 (testcrawler@hotmail.com)
Validator.nu/LV
Validator.nu/LV http://validator.w3.org/services
VeBot (+http://www.veinteractive.com/vebot)
VegeBot (we follow your robots.txt settings before crawling, you can slow down the bot by change the Crawl-Delay parameter in the settings.if you have an enquiry, please email to: abuse-report@terrykyleseoagency.com)
Vegi bot (we follow your robots.txt settings before crawling, you can slow down the bot by change the Crawl-Delay parameter in the settings.if you have an enquiry, please email to: abuse-report@terrykyleseoagency.com)
VelenPublicWebCrawler (velen.io)
Vercelbot (+https://vercel.com)
Viber
VidibleScraper/1.0
VinjaVideoSpider/1.1
Virusdie crawler/2.1
VisBot/2.0 (Visvo.com Crawler; http://www.visvo.com/bot.html; bot@visvo.com)
Visbot/2.0 (+http://www.visvo.com/en/webmasters.jsp;bot@visvo.com)
Visited by http://tools.geek-tools.org
Vivante Link Checker (http://www.vivante.com)
Vodafone mCrawler (bergum@fast.no)
Vorboss Web Crawler [crawl@vorboss.net]/Nutch-2.3
Vortex/2.2 ( http://marty.anstey.ca/robots/vortex/)
Vortex/2.2 (+http://marty.anstey.ca/robots/vortex/)
Vulnbusters Meter (see http://vulnbusters.com for details)
W3C-checklink
W3C-checklink/2.90 libwww-perl/5.64
W3C-checklink/3.6.2.3 libwww-perl/5.64
W3C-checklink/4.0 [4.4] libwww-perl/5.803
W3C-checklink/4.2 [4.20] libwww-perl/5.803
W3C-checklink/4.2.1 [4.21] libwww-perl/5.803
W3C-checklink/4.3 [4.42] libwww-perl/5.805
W3C-checklink/4.3 [4.42] libwww-perl/5.808
W3C-checklink/4.3 [4.42] libwww-perl/5.820
W3C-checklink/4.5 [4.154] libwww-perl/5.823
W3C-checklink/4.5 [4.160] libwww-perl/5.823
W3C-checklink/4.81 libwww-perl/5.836
W3C-mobileOK/DDC-1.0
W3C-mobileOK/DDC-1.0 (see http://www.w3.org/2006/07/mobileok-ddc)
W3C_I18n-Checker/1.0
W3C_I18n-Checker/1.0 (http://validator.w3.org/services)
W3C_Unicorn/1.0
W3C_Unicorn/1.0 (http://validator.w3.org/services)
W3C_Validator/1.3
W3C_Validator/1.3 http://validator.w3.org/services
W3C_Validator/1.767
WEBB Crawler - see: http://badcheese.com/robots.html
WEPA/3.1 (http://www.wepa.com/; webmaster@wepa.com)
WGETbot/1.0 (+http://wget.alanreed.org)
WIRE/0.11 (Linux; i686; Robot,Spider,Crawler,aromano@cli.di.unipi.it)
WIRE/1.0 (Linux;i686;Bot,Robot,Spider,Crawler)
WMCAI-robot (http://www.topicmaster.jp/wmcai/crawler.html)
WNMbot/0.4.0
WPScan v2.9 (http://wpscan.org)
WSDL Crawler
WSDLSpider/1.0 (http://www.wsdlworld.com)
WWW::LayeredExtractor::Handler::Feed/0.01
WatchMouse/18990 (http://watchmouse.com/ ; bc.watchmouse.com)
WatchMouse/18990 (http://watchmouse.com/ ; d2.watchmouse.com)
WatchMouse/18990 (http://watchmouse.com/ ; d3.watchmouse.com)
WatchMouse/18990 (http://watchmouse.com/ ; gab)
WatchMouse/18990 (http://watchmouse.com/ ; hk)
WatchMouse/18990 (http://watchmouse.com/ ; it)
WatchMouse/18990 (http://watchmouse.com/ ; liz)
WatchMouse/18990 (http://watchmouse.com/ ; ny)
WatchMouse/18990 (http://watchmouse.com/ ; se.watchmouse.com)
WatchMouse/18990 (http://watchmouse.com/ ; uk)
WatchMouse/8.4.0.3 (http://watchmouse.com/ ; chzrh01.watchmouse.net)
WatchMouse/8.4.0.3 (http://watchmouse.com/ ; decgn02.watchmouse.net)
WatchMouse/8.4.0.3 (http://watchmouse.com/ ; demuc02.watchmouse.net)
WatchMouse/8.4.0.3 (http://watchmouse.com/ ; gblon01.watchmouse.net)
WatchMouse/8.4.0.3 (http://watchmouse.com/ ; grath01.watchmouse.net)
WatchMouse/8.4.0.3 (http://watchmouse.com/ ; hkhkg02.watchmouse.net)
WatchMouse/8.4.0.3 (http://watchmouse.com/ ; itmil01.watchmouse.net)
WatchMouse/8.4.0.3 (http://watchmouse.com/ ; itpda01.watchmouse.net)
WatchMouse/8.4.0.3 (http://watchmouse.com/ ; lvps91-250-96-109.dedicated.hosteurope.de)
WatchMouse/8.4.0.3 (http://watchmouse.com/ ; plgdn01.watchmouse.net)
WatchMouse/8.4.0.3 (http://watchmouse.com/ ; sesto02.watchmouse.net)
WatchMouse/8.4.0.3 (http://watchmouse.com/ ; uschi02.watchmouse.net)
WatchMouse/8.4.0.3 (http://watchmouse.com/ ; usdal02.watchmouse.net)
WeSEE:Ads/PageBot (http://www.wesee.com/bot/)
WeSEE:Ads/PictureBot (http://www.wesee.com/bot/)
WeSEE:Search
WeSEE:Search/0.1 (Alpha, http://www.wesee.com/bot/)
WeSEE:Search/0.1 (Alpha, http://www.wesee.com/en/support/bot/)
WeSEE_Bot:we_help_monitize_your_site (http://www.wesee.com/bot/)
Web-Robot/5.0 (en-US; web-robot.com/policy.html) Web-Robot Crawler/2.0.3
Web-sniffer.me/1.0.0 (+http://web-sniffer.me/)
Web-sniffer/1.0.31 (+http://web-sniffer.net/)
Web-sniffer/1.1.0 (+http://web-sniffer.net/)
WebAlta Crawler/2.0 (http://www.webalta.net/ru/about_webmaster.html) (Windows; U; Windows NT 5.1; ru-RU)
WebCompanyCrawler
WebCookies/1.0 (+http://webcookies.info/faq/#agent)
WebCrawler v1.3
WebGazer/1.0 (+https://www.webgazer.io)
WebImages 0.3 ( http://herbert.groot.jebbink.nl/?app=WebImages )
WebRankSpider/1.37 (+http://ulm191.server4you.de/crawler/)
WebReaper v9.7 - www.webreaper.net
WebRingChecker/1.71 LWP/5.79
WebSniffer/1.0 (+http://websniffer.cc/)
WebTarantula.com Crawler
WebVulnCrawl.blogspot.com/1.0 libwww-perl/5.803
WebWatch/Robot_txtChecker
WebZIP/7.0 (http://www.spidersoft.com)
WebarooBot (Webaroo Bot; http://64.124.122.252/feedback.html)
WebbCrawler 1.0 ( http://badcheese.com/crawler.html )
Webshot/0.9 ( http://www.shrinktheweb.com/ )
WebsiteExtractor
Webspider/1.0 (web spider; )
Websquash.com (Add url robot)
WellKnownBot/0.1 (https://well-known.dev)
Wget/1.14 (linux-gnu)
Wget/1.15 (linux-gnu)
Wget/1.20.3 (linux-gnu)
WhatsApp/0.3.4479 N
WhatsApp/0.3.4679 N
WhatsApp/0.3.4941 N
WhatsApp/2.12.449 A
WhatsApp/2.12.453 A
WhatsApp/2.12.510 A
WhatsApp/2.12.540 A
WhatsApp/2.12.548 A
WhatsApp/2.12.555 A
WhatsApp/2.12.556 A
WhatsApp/2.16.13 A
WhatsApp/2.16.42 A
WhatsApp/2.16.57 A
WhatsApp/2.19.175 A
WhatsApp/2.19.244 A
WhatsApp/2.19.258 A
WhatsApp/2.19.308 A
WhatsApp/2.19.330 A
Who.is Bot
Whoismindbot/1.0 (+http://www.whoismind.com/bot.html)
WikiDo/1.1 (http://wikido.com; crawler@wikido.com)
WikioFeedBot 1.0 (http://www.wikio.com)
Willow Internet Crawler by Twotrees V2.1
WillyBot/1.1 (http://www.willyfogg.com/info/willybot)
WinPodder (http://winpodder.com)
WinWebBot/1.0; (Balaena Ltd, UK); http://www.balaena.com/winwebbot.html; winwebbot@balaena.com;)
Windows-RSS-Platform/2.0 (IE 11.0; Windows NT 10.0)
Windows-RSS-Platform/2.0 (MSIE 9.0; Windows NT 6.0)
Winds: Open Source RSS & Podcast app: https://getstream.io/winds/
WinkBot/0.06 (Wink.com search engine web crawler; http://www.wink.com/Wink:WinkBot; winkbot@wink.com)
WoFindeIch Robot 1.0(+http://www.search.wofindeich.com/robot.php)
WocBot/Mozilla/5.0 (Wocodi Web Crawler 1.0; http://www.wocodi.com/crawler; crawler@wocodi.com)
Woko robot 3.0
Wolfram HTTPClient 12.
WomlpeFactory/0.1 (+http://www.Womple.com/bot.html)
WordPress.com mShots; http://support.wordpress.com/contact/
WordPress.com; http://fleurdelyz.com
WordPress.com; https://botd.wordpress.com
WordPress.com; https://public-api.wordpress.com
WordPress.com; https://ramaransonvsthecult.wordpress.com
WordPress/4.0.1; http://bestcomp.hol.es
WordPress/4.3.3; http://homebusinessreviews.online
WordPress/4.3.3; http://legithomejobs.club
WordPress/4.4.2; http://cheapautoinsuranceinva.com
WordPress/4.4.2; http://citycollegeinmiami.com
WordPress/4.4.2; http://cookingideabook.com
WordPress/4.4.2; http://oldmanhealth.com
WordPress/4.4.2; http://sapcrmarchitecture.com
WordPress/4.4.2; http://www.spar3d.com
WordPress/4.5.2; http://1.we-careu.xyz
WordPress/4.7.2; https://example.com
WordPress/x.x.x.x PHP/4.x.xxMozilla/5.0 (Unknown; Linux x86_64) AppleWebKit/534.34 (KHTML, like Gecko) PhantomJS/1.6.1 Safari/534.34
WordupInfoSearch/1.0
WordupinfoSearch/1.0
WorldBrewBot/2.1 (+http://www.marketbrew.com/)
Wotbox/2.0 (bot@wotbox.com; http://www.wotbox.com)
Wotbox/2.01 ( http://www.wotbox.com/bot/)
Wotbox/2.01 (+http://www.wotbox.com/bot/)
X-Crawler
XRL/3.00 (Linux; i686; en-us) (+http://metamark.net/about)
XSpider 0.01;http://uncool.oicp.net/spider.html
XYZ Spider
Xaldon WebSpider 2.7.b6
Xaldon_WebSpider/2.0.b1
XenForo/2.x (https://www.example.com)
XenForo/2.x (https://www.recreationalflying.com)
Xenu Link Sleuth 1.2b
Xenu Link Sleuth 1.2c
Xenu Link Sleuth 1.2d
Xenu Link Sleuth 1.2e
Xenu Link Sleuth 1.2f
Xenu Link Sleuth 1.2g
Xenu Link Sleuth 1.2h
Xenu Link Sleuth 1.2i
Xenu Link Sleuth/1.3.7
Xenu Link Sleuth/1.3.8
Xerka WebBot v1.0.0 [AIDO_CREA_ab]
XmlSitemapGenerator - http://XmlSitemapGenerator.org
XmlSitemapGenerator - http://xmlsitemapgenerator.org
XoviBot/1.0
Y!J-ASR/0.1 crawler (http://www.yahoo-help.jp/app/answers/detail/p/595/a_id/42716/)
Y!J-BRI/0.0.1 crawler ( http://help.yahoo.co.jp/help/jp/search/indexing/indexing-15.html )
Y!J-BRJ/YATS crawler (http://help.yahoo.co.jp/help/jp/search/indexing/indexing-15.html)
Y!J-BRJ/YATS crawler (http://listing.yahoo.co.jp/support/faq/int/other/other_001.html)
Y!J-BRO/YFSJ crawler (compatible; Mozilla 4.0; MSIE 5.5; http://help.yahoo.co.jp/help/jp/search/indexing/indexing-15.html; YahooFeedSeekerJp/2.0)
Y!J-BRO/YFSJ crawler (compatible; Mozilla 4.0; MSIE 5.5; http://help.yahoo.co.jp/help/jp/search/indexing/indexing-15.html; YahooFeedSeekerJp/2.0; users 0; views 248)
Y!J-BRP/YFSBJ crawler (compatible; Mozilla 4.0; MSIE 5.5; http://help.yahoo.co.jp/help/jp/search/indexing/indexing-15.html; YahooFeedSeekerBetaJp/2.0; users 0; views 80)
Y!J-BRW/1.0 (https://www.yahoo-help.jp/app/answers/detail/p/595/a_id/42716)
Y!J-BRW/1.0 crawler (http://help.yahoo.co.jp/help/jp/search/indexing/indexing-15.html)
Y!J-BSC/1.0 (http://help.yahoo.co.jp/help/jp/blog-search/)
Y!J-BSC/1.0 crawler (http://help.yahoo.co.jp/help/jp/blog-search/)
Y!J-PSC/1.0 crawler (http://help.yahoo.co.jp/help/jp/search/indexing/indexing-15.html)
YOOBI - WebBot 1.0
YOOBI.de - WebBot 1.0
YOURLS v1.5.1 +http://yourls.org/ (running on http://fhort.com)
YOURLS v1.7 +http://yourls.org/ (running on http://shortr.at)
YOURLS v1.7.1 +http://yourls.org/ (running on http://flnx.co)
YRL_ODP_CRAWLER
Yaanb/1.5.001 (compatible; Win64;+http://www.yaanb.com/company/bot.hmtl)
Yahoo! Site Explorer Feed Validator http://help.yahoo.com/l/us/yahoo/search/siteexplorer/manage/
Yahoo-MMCrawler/3.x (mm dash crawler at trd dot overture dot com)
Yahoo-Newscrawler/3.9 (news-search-crawler at yahoo-inc dot com)
Yahoo-VerticalCrawler-FormerWebCrawler/3.9 crawler at trd dot overture dot com; http://www.alltheweb.com/help/webmaster/crawler
Yahoo:LinkExpander:Slingstone
YahooCacheSystem
YahooMailProxy; https://help.yahoo.com/kb/yahoo-mail-proxy-SLN28749.html
YahooSeeker-Testing/v3.9 (compatible; Mozilla 4.0; MSIE 5.5; http://search.yahoo.com/)
YahooSeeker/1.0 (compatible; Mozilla 4.0; MSIE 5.5; http://help.yahoo.com/help/us/shop/merchant/)
Yanga WorldSearch Bot v1.1/beta (http://www.yanga.co.uk/)
Yasaklibot/v1.2 (http://www.Yasakli.com/bot.php)
YebolBot (Email: yebolbot@gmail.com; If the web crawling affects your web service, or you don't like to be crawled by us, please email us. We'll stop crawling immediately.)
Yepi/1.0 (NHN Corp.; http://help.naver.com/robots/)
Yeti-FeedItemCrawler/1.0 (NHN Corp.; http://help.naver.com/robots/)
Yeti/1.0 (+http://help.naver.com/robots/)
Yeti/1.0 (NHN Corp.; http://help.naver.com/robots/)
Yeti/1.1 (NHN Corp.; http://help.naver.com/robots/)
Yeti/1.1 (Naver Corp.; http://help.naver.com/robots/)
YisouSpider
YoonoCrawler/1.0 (crawler@yoono.com)
YottaBot Spider/Nutch-2.3
Your Nutch Spider/Nutch-2.2.1
YowedoBot/Yowedo 1.0 (Search Engine crawler for yowedo.com; http://yowedo.com/en/partners.html; crawler@yowedo.com)
ZIBB Crawler (email address / WWW address)
Zabbix
Zao-Crawler
Zao/0.1 (http://www.kototoi.org/zao/)
Zao/0.1 (http://www.kototol.org/zao)
ZeBot_www.ze.bz (ze.bz@hotmail.com)
Zemanta Aggregator/0.9 +http://www.zemanta.com
Zeusbot/0.8.1 (Ulysseek's web-crawling robot; http://www.zeusbot.com; agent@zeusbot.com)
ZillaCrawler
ZnajdzFoto/ImageBot 2.0b
ZooShot 0.42
Zookabot/2.0;++http://zookabot.com
Zookabot/2.1;++http://zookabot.com
Zookabot/2.2;++http://zookabot.com
Zookabot/2.4;++http://zookabot.com
Zookabot/2.5;++http://zookabot.com
ZoomBot (Linkbot 1.0 http://suite.seozoom.it/bot.html)
ZoomInfo::Beehive Crawler
ZoomInformation Bot
ZoomSpider - wrensoft.com
ZoominfoBot (zoominfobot at zoominfo dot com)
ZumBot/1.0 (ZUM Search; http://help.zum.com/inquiry)
^Nail (http://CaretNail.com)
a.pr-cy.ru
abot/0.1 (abot; http://www.abot.com; abot@abot.com)
acebookexternalhit/1.0 (+http://www.facebook.com/externalhit_uatext.php)
acquia-crawler (detected bad behaviour? please tell us at it@acquia.com),gzip(gfe),gzip(gfe)
adbeat_bot
adidxbot/1.1 (+http://search.msn.com/msnbot.htm)
adidxbot/2.0 (+http://search.msn.com/msnbot.htm)
agbot/1.0 (AgHaven.com search engine crawler; http://search.aghaven.com; bot@aghaven.com)
agentslug.com - website monitoring tool
al_org_viewer (larbin2.6.3@unspecified.mail)
al_viewer (larbin2.6.3@unspecified.mail)
alexa v0.1.4 (http://www.openwebspider.org/)
all_web1 (larbin2.6.3@unspecified.mail)
amibot - http://www.amidalla.de - tech@amidalla.com libwww-perl/5.831
amphetameme crawler (crawler@amphetameme.com)
annotate_google; http://ponderer.org/download/annotate_google.user.js
anonymous coward/1.0 (a Storm-based crawler; https://github.com/DigitalPebble/storm-crawler; someone@company.com)
ant.com/Nutch-1.7 (http://ant.com)
argenfybot
asked/Nutch-0.8 (web crawler; http://asked.jp; epicurus at gmail dot com)
audioCrawlerBot/1.0 (http://www.audiocrawler.com/)
audisto.com full crawler 3.26.431 (refer to in robots.txt as audisto, see https://audisto.com/bot)
audisto.com full crawler 4.33.470 (refer to in robots.txt as audisto, see https://audisto.com/bot)
awesomebar_scraper/1.0
axadine/ (Axadine Crawler; http://www.axada.de/; )
axios/0.17.1
axios/0.18.0
axios/0.19.0
baiduspider-mobile-gate
bandscraper
baypup/1.1 (Baypup; http://www.baypup.com/; jason@baypup.com)
baypup/colbert (Baypup; http://sf.baypup.com/webmasters; jason@baypup.com)
baypup/colbert (Baypup; http://www.baypup.com/webmasters; jason@baypup.com)
bidswitchbot/1.0
binlar_2.6.3 (larbin2.6.3@unspecified.mail)
binlar_2.6.3 binlar2.6.3@unspecified.mail
binlar_2.6.3 binlar_2.6.3@unspecified.mail
binlar_2.6.3 larbin2.6.3@unspecified.mail
binlar_2.6.3 phanendra_kalapala@McAfee.com
binlar_2.6.3 test@mgmt.mic
bitlybot
bitlybot/2.0
bitlybot/3.0
bitlybot/3.0 (+http://bit.ly/)
bl.uk_lddc_bot/3.1.1 (+http://www.bl.uk/aboutus/legaldeposit/websites/websites/faqswebmaster/index.html)
bl.uk_lddc_bot/3.3.0-LBS-2016-02 (+http://www.bl.uk/aboutus/legaldeposit/websites/websites/faqswebmaster/index.html)
bl.uk_lddc_bot/3.3.0-SNAPSHOT-2014-10-07T09:33:31Z (+http://www.bl.uk/aboutus/legaldeposit/websites/websites/faqswebmaster/index.html)
bl.uk_lddc_bot/3.3.0-SNAPSHOT-20140519-0844 (+http://www.bl.uk/aboutus/legaldeposit/websites/websites/faqswebmaster/index.html)
blackspider
blog_crawler/1.0
blogmuraBot (+http://www.blogmura.com)
boitho crawler
boitho.com-dc/0.4 ( http://www.boitho.com/dcbot.html )
boitho.com-dc/0.79 ( http://www.boitho.com/dcbot.html )
boitho.com-dc/0.81 ( http://www.boitho.com/dcbot.html )
boitho.com-dc/0.82 ( http://www.boitho.com/dcbot.html )
boitho.com-dc/0.83 ( http://www.boitho.com/dcbot.html )
boitho.com-dc/0.85 ( http://www.boitho.com/dcbot.html )
boitho.com-dc/0.86 ( http://www.boitho.com/dcbot.html )
boitho.com-robot/1.0
boitho.com-robot/1.1
bot-pge.chlooe.com/1.0.0 (+http://www.chlooe.com/)
bot.wsowner.com/1.0.0 (+http://wsowner.com/)
botrobin/Nutch-1.13-SNAPSHOT (http://smarter.codes/bot-robin/; botrobin@smarter.codes)
cXensebot/1.1a
calculon spider/0.1.0 mailto:abuse@yieldbot.com
camoca/camoca-n.1.2 (super-agent; search crawler; info at does not exist dot com)
cb crawl (+http://www.companybooknetworking.com)
check_http/v1.4.16 (nagios-plugins 1.4.16)
check_http/v1.5 (nagios-plugins 1.5)
check_http/v2.0.3 (nagios-plugins 2.0.3)
check_http/v2.2.1 (nagios-plugins 2.2.1)
checkgzipcompression.com robot
cis455crawler
classbot (+http://allclasses.com)
cloudforecastbot
cn_ab_viewer (larbin2.6.3@unspecified.mail)
coccoc
coccoc/1.0 ()
coccoc/1.0 (http://help.coccoc.com/)
coccoc/1.0 (http://help.coccoc.vn/)
coccocbot-web/1.0 (+http://help.coccoc.com/searchengine)
collectd/5.5.1
colly - https://github.com/gocolly/colly
colly - https://github.com/gocolly/colly/v2
com.hubspot.spooks.core.UserAgent@753535c0, HubSpot Marketing Grader
compatible; Licorne Uri Checker 1.0; +http://www.jednorozec.cz)
compatible; Netseer crawler/2.0; +http://www.netseer.com/crawler.html; crawler@netseer.com
copyright sheriff (+http://www.copyrightsheriff.com/)
cosmos/0.9_(robot@xyleme.com)
crawler for netopian (http://www.netopian.co.uk/)
crawler4j (http://code.google.com/p/crawler4j/)
crawler4j (https://github.com/yasserg/crawler4j/)
crawler4j for XQuery
crazywebcrawler 0.9.0, http://www.crazywebcrawler.com
curl
curl/7.16.0 (i386-unknown-openbsd4.1) libcurl/7.16.0 OpenSSL/0.9.7j zlib/1.2.3
curl/7.29.0
curl/7.47.0
curl/7.54.0
curl/7.55.1
curl/7.64.0
curl/7.64.1
curl/7.65.3
cuwhois/1.0 (+http://www.cuwhois.com/)
dCrawlBot/1.0.1120
datagnionbot (+http://www.datagnion.com/bot.html)
dcrawl/1.0
deepcrawler/3.1 (http://www.queusearch.com/whatis_deepcrawler.php),gzip(gfe),gzip(gfe)
deeris/1.0 (+http://www.deeris.com)
dj-research/Nutch-1.11 (analytics@@demandjump.com)
dlvr.it/1.0 (+http://dlvr.it/)
dlvr.it/1.0 (+http://dlvr.it/) Mozilla/5.0
domainsbot (+http://www.domainsbot.com)
dotbot
downnotifier.com monitoring
drupact/0.7; http://www.arocom.de/drupact
dtSearchSpider
e.ventures Investment Crawler (eventures.vc)
eBot / v.0.1e (http://alfa.elchron.cz)
eBot / v.1.0a (http://alfa.elchron.cz)
eCairn-Grabber/1.0 (+http://ecairn.com/grabber) curl/7.15
eCommerceBot (http://www.ehandel.se/botinfo.html)
eZ Publish Link Validator
ec2linkfinder
elefent/1.2 (Web Crawler; https://crawler.elefent.net; webmaster at elefent dot net)
elefent/Elefent 1.1 (Friendly web walker.; http://elefent.eu/; jason@elefent.eu)
elefent/Elefent 1.1 (Friendly web walker.; http://www.elefent.eu/; jason@elefent.eu)
elefent/Elefent 1.2 (A friendly web elefent.; http://elefent.eu/; webmaster@elefent.eu)
emBot-GalaBuzz/Nutch-1.0 (http://emining.jp/; em@galabuzz.jp)
emefgebot/beta (+http://emefge.de/bot.html)
endo/1.0 (Mac OS X; ppc i386; http://kula.jp/endo)
enicura crawler 1.0,gzip(gfe),gzip(gfe)
enlle punto com/Nutch-1.9
envolk/1.7 (+http://www.envolk.com/envolkspiderinfo.html)
envolk[ITS]spider/1.6 ( http://www.envolk.com/envolkspider.html)
es_com_viewer (larbin2.6.3@unspecified.mail)
eseek-crawler.0.5 (crawler@exactseek.com)
everyfeed-spider/2.0 (http://www.everyfeed.com)
exooba/exooba crawler (exooba; exooba)
facebookexternalhit/1.0 (+http://www.facebook.com/externalhit_uatext.php)
facebookexternalhit/1.1
facebookexternalhit/1.1 (+http(s)://www.facebook.com/externalhit_uatext.php)
facebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php)
facebookplatform/1.0 (+http://developers.facebook.com)
facebookscraper/1.0( http://www.facebook.com/sharescraper_help.php)
fastbot crawler beta 2.0 (+http://www.fastbot.de)
fastbot crawler beta 4.0 (+http://www.fastbot.de)
fastlwspider/1.0
faviconkit/1.0 (https://faviconkit.com)
feedzirra http://github.com/pauldix/feedzirra/tree/master
findlinks/1.0 (+http://wortschatz.uni-leipzig.de/findlinks/)
findlinks/1.1.3-beta8 (+http://wortschatz.uni-leipzig.de/findlinks/)
findlinks/1.1.3-beta9 (+http://wortschatz.uni-leipzig.de/findlinks/)
findlinks/1.1.5-beta7 (+http://wortschatz.uni-leipzig.de/findlinks/)
findlinks/1.1.6-beta1 (+http://wortschatz.uni-leipzig.de/findlinks/)
findlinks/1.1.6-beta1 (+http://wortschatz.uni-leipzig.de/findlinks/; YaCy 0.1; yacy.net)
findlinks/1.1.6-beta2 (+http://wortschatz.uni-leipzig.de/findlinks/)
findlinks/1.1.6-beta3 (+http://wortschatz.uni-leipzig.de/findlinks/)
findlinks/1.1.6-beta4 (+http://wortschatz.uni-leipzig.de/findlinks/)
findlinks/1.1.6-beta5 (+http://wortschatz.uni-leipzig.de/findlinks/)
findlinks/1.1.6-beta6 (+http://wortschatz.uni-leipzig.de/findlinks/)
findlinks/2.0 (+http://wortschatz.uni-leipzig.de/findlinks/)
findlinks/2.0.1 (+http://wortschatz.uni-leipzig.de/findlinks/)
findlinks/2.0.2 (+http://wortschatz.uni-leipzig.de/findlinks/)
findlinks/2.0.4 (+http://wortschatz.uni-leipzig.de/findlinks/)
findlinks/2.0.5 (+http://wortschatz.uni-leipzig.de/findlinks/)
findlinks/2.0.9 (+http://wortschatz.uni-leipzig.de/findlinks/)
findlinks/2.1 (+http://wortschatz.uni-leipzig.de/findlinks/)
findlinks/2.1.3 (+http://wortschatz.uni-leipzig.de/findlinks/)
findlinks/2.1.5 (+http://wortschatz.uni-leipzig.de/findlinks/)
findlinks/2.2 (+http://wortschatz.uni-leipzig.de/findlinks/)
findlinks/2.5 (+http://wortschatz.uni-leipzig.de/findlinks/)
findlinks/2.6 ( http://wortschatz.uni-leipzig.de/findlinks/)
findlinks/2.6 (+http://wortschatz.uni-leipzig.de/findlinks/)
flatlandbot/allspark (Flatland Industries Web Spider; http://www.flatlandindustries.com/flatlandbot; jason@flatlandindustries.com)
flatlandbot/baypup (Flatland Industries Web Spider; http://www.flatlandindustries.com/flatlandbot; jason@flatlandindustries.com)
flieder - neofonie heritrix/1.14.3 (+http://spider.neofonie.de)
focusbot/1.1
freshrss/0.8-dev (Linux; http://freshrss.org) SimplePie/1.4-dev-FreshRSS
fuelbot
g2Crawler nobody@airmail.net
g2reader-bot/1.0 (+http://www.g2reader.com/)
gMozilla/5.0 (compatible; Infohelfer/1.4.3; +http://www.infohelfer.de/crawler.php)
ganarvisitas/1.0 (+http://www.ganarvisitas.com/)
genieBot ((http://64.5.245.11/faq/faq.html))
gimme60 (Gimme60 Store ID Bot; gimme60.com)
gipo-crawler/Nutch-1.10 (Global Internet Policy Observatory crawler)
gobuster/3.1.0
gonzo/1[P] (+http://www.suchen.de/faq.html)
gonzo1[D] mailto:crawleradmin.t-info@telekom.de
gonzo1[P] +http://www.suchen.de/faq.html
gonzo2[P] +http://www.suchen.de/faq.html
gonzo2[p] (+http://www.suchen.de/faq.html)
gooblog/2.0 (http://help.goo.ne.jp/contact/)
gooblogsearch/2.0 (http://search.goo.ne.jp/option/use/sub4/sub4-1/)
google-xrawler
grub-client-1.5.3; (grub-client-1.5.3; Crawl your own stuff with http://grub.org)
gsa-crawler
haupia-crawler
hawkReader/0.9b (Feed Parser; http://www.hawkreader.com; Allow like Gecko)
hawkReader/1.8 (Link Parser; http://www.hawkreader.com/; Allow like Gecko) Build/f2b2566
help@dataminr.com
hitcrawler_0.1 (ringken@gmail.com)
hivaBot/hivaBot-0.2 (info@gorgor.ir)
hivaBot/hivaBot-1.0 (Iranian Search Engine bot; http://yooz.ir; info@yooz.ir)
hl_ftien_spider
hl_ftien_spider_v1.1
holmes/3.12.4 (http://morfeo.centrum.cz/bot)
http-request/v0.7.0 (http://git.io/tl_S2w) node.js/v0.10.29
http.rb/2.2.2 (Mastodon/1.5.1; +https://example-masto-instance.org/)
http.rb/2.2.2 (Mastodon/1.6.1; +https://mathtod.online/)
http.rb/3.0.0 (Mastodon/2.2.0; +https://mstdn.io/)
http.rb/3.2.0 (Mastodon/2.4.3; +https://uwu.social/)
http://arachnode.net 1.2
http://arachnode.net 1.4
http://arachnode.net 2.5
http://domino.research.ibm.com/comm/research_projects.nsf/pages/sai-crawler.callingcard.html
http://seewithkids.com/bot
http://tools.geek-tools.org/link-counter/
http://www.Syntryx.com/ ANT Chassis 9.27; Mozilla/4.0 compatible crawler
http://www.almaden.ibm.com/cs/crawler
http://www.almaden.ibm.com/cs/crawler [wf84]
http://www.amagit.com/
http://www.checkprivacy.or.kr:6600/RS/PRIVACY_ENFAQ.jsp
http://www.mabontland.com
http://www.nominet.org.uk/privacypolicy
http://www.uni-koblenz.de/~flocke/robot-info.txt
http_get
https://gdnplus.com:Gather Analyze Provide.
https://getroot.sh survey
https://www.nominet.uk/privacy-notice
https_page_test (larbin2.6.3@unspecified.mail)
httpunit/1.x
httpx - Open-source project (github.com/projectdiscovery/httpx)
i1searchbot/2.0 (i1search web crawler; http://www.i1search.com; crawler@i1search.com)
iCCrawler (http://www.iccenter.net/bot.htm)
ia_archiver (+http://www.alexa.com/site/help/webmasters; crawler@alexa.com)
ia_archiver-web.archive.org
ia_archiver/8.7 (Windows NT 5.0; )
ias-va/3.1 (+https://www.admantx.com/service-fetcher.html)
iaskspider/2.0(+http://iask.com/help/help_index.html)
iaskspider2 (iask@staff.sina.com.cn)
ichiro/1.0 (ichiro@nttr.co.jp)
ichiro/2.0 (http://help.goo.ne.jp/door/crawler.html)
ichiro/2.0 (ichiro@nttr.co.jp)
ichiro/3.0 (http://help.goo.ne.jp/door/crawler.html)
ichiro/3.0 (http://help.goo.ne.jp/help/article/1142)
ichiro/3.0 (http://search.goo.ne.jp/option/use/sub4/sub4-1/)
ichiro/4.0 (http://help.goo.ne.jp/door/crawler.html)
ichiro/5.0 (http://help.goo.ne.jp/door/crawler.html)
igdeSpyder (compatible; igde.ru; +http://igde.ru/doc/tech.html)
iisbot/1.0 (+http://www.iis.net/iisbot.html)
inboundli link checker
infohelfer/1.1.0 (http://www.infohelfer.de/)
infoobot/0.1 (https://www.infoo.nl/bot.html)
ip-web-crawler.com
iqdb/0.1 (+http://iqdb.org/)
iqdb/0.2 (+http://iqdb.harry.lu/)
isUp.li Website Monitoring 2
iskanie (+http://www.iskanie.com)
istellabot-nutch/Nutch-1.10
istellabot/Nutch-1.10
istellabot/Nutch-1.11
istellabot/t.1
it2media-domain-crawler/1.0 on crawler-prod.it2media.de
it2media-domain-crawler/2.0
jetmon/1.0 (Jetpack Site Uptime Monitor by WordPress.com)
jpg-newsbot/2.0; (+https://vipnytt.no/bots/)
jrCrawler/1.0b
k6/0.31.1 (https://k6.io/)
kalooga/KaloogaBot (Kalooga; http://www.kalooga.com/info.html?page=crawler)
kizasi-spider/1.0 (+http://kizasi.jp/)
kouio.com RSS reader
kouio.com RSS reader - 6 subscribers
larbin_2.5.0 (larbin2.5.0@unspecified.mail)
larbin_2.6.2 kalou@kalou.net
larbin_2.6.2 pierre@micro-fun.ch
larbin_2.6.3 (atyzos@yahoo.com)
larbin_2.6.3 (larbin2.6.3@unspecified.mail)
larbin_2.6.3 capveg@cs.umd.edu
larbin_2.6.3 gqnmgsp@ruc.edu.cn
larbin_2.6.3 larbin2.6.3@unspecified.mail
larbin_2.6.4 (atyzos@yahoo.com)
lb-spider
ldspider (BTC 2011 crawl, harth@kit.edu, http://code.google.com/p/ldspider/wiki/Robots)
ldspider (http://code.google.com/p/ldspider/wiki/Robots)
li_viewer (larbin2.6.3@unspecified.mail)
librabot/1.0 (+http://search.msn.com/msnbot.htm)
librabot/2.0 (+http://academic.research.microsoft.com/)
librabot/2.0 (+http://search.msn.com/msnbot.htm)
libwww-perl/5.53
libwww-perl/5.800 (+http://passoire.afraid.org/mylittlewebsurvey/index.html)
linguatools-bot/Nutch-1.6 (searching for translated pages; http://www.linguatools.de/linguatoolsbot.html; peter dot kolb at linguatools dot org)
linkapediabot (+http://www.linkapedia.com)
linkdex.com/v2.0
linkdex.com/v2.0 and linkdex.com/v2.1
linkdexbot/Nutch-1.0-dev (http://www.linkdex.com/; crawl at linkdex dot com)
livedoor FeedFetcher/0.01 (http://reader.livedoor.com/; 999 subscribers)
lmspider (lmspider@scansoft.com)
ltx71 - (http://ltx71.com/)
lua-resty-http/0.10 (Lua) ngx_lua/10000
lworldspider
magpie-crawler/1.1 (U; Linux amd64; en-GB; +http://www.brandwatch.net)
mahonie, neofonie search:robot/search:robot/0.0.1 (This is the MIA Bot - crawling for mia research project. If you feel unhappy and do not want to be visited by our crawler send an email to spider@neofonie.de; http://spider.neofonie.de; spider@neofonie.de)
makecontact/1.0 (+https://makecontact.io)
mapion-news-bot/1.0 (http://www.mapion.co.jp/news/)
masidani_bot_v0.3 (info@masidani.com)
masscan/1.0 (https://github.com/robertdavidgraham/masscan)
mbot v.1.16
media-percbotspider <ops@percolate.com>
mfibot/1.1 (http://www.mfisoft.ru/analyst/; <admin@mfisoft.ru>; en-RU)
mindUpBot
mindUpBot (datenbutler.de)
mixi-check/1.0 (http://mixi.jp/)
mixi-crawler/2.00 (http://mixi.jp/)
mixi-news-crawler/1.00 (http://mixi.jp/)
montastic-monitor http://www.montastic.com
mozDex/0.05-dev (mozDex; http://www.mozdex.com/bot.html; spider@mozdex.com)
mozilla/5.0 (compatible; discobot/1.1; +http://discoveryengine.com/discobot.html)
mozilla/5.0 (compatible; heritrix/1.3.0 http://archive.crawler.org)
mozilla/5.0 (compatible; webmastercoffee/0.7; +http://webmastercoffee.com/about)
mozilla/5.0 (larbin2.6.3@unspecified.mail)
msnbot-NewsBlogs/2.0b (+http://search.msn.com/msnbot.htm)
msnbot-UDiscovery/2.0b (+http://search.msn.com/msnbot.htm)
msnbot-media/1.0 ( http://search.msn.com/msnbot.htm)
msnbot-media/1.0 (+http://search.msn.com/msnbot.htm)
msnbot-media/1.1 (+http://search.msn.com/msnbot.htm)
msnbot-media/2.0b (+http://search.msn.com/msnbot.htm)
msnbot/1.0 (+http://search.msn.com/msnbot.htm)
msnbot/1.1 (+http://search.msn.com/msnbot.htm)
msnbot/2.0b (+http://search.msn.com/msnbot.htm)
msnbot/2.0b (+http://search.msn.com/msnbot.htm).
msnbot/2.0b (+http://search.msn.com/msnbot.htm)._
msnbot/2.0b v1394720947.6703
munin/2.0.30-1 (libwww-perl/6.15)
n4p_bot (crawler@n4p.com)
nWormFeedFinder (http://www.nworm.com)
na-Webcrawler (helpdesk@newsaktuell.de)
nerdybot
netEstate Impressumscrawler (+http://www.netestate.de/De/Loesungen/Impressumscrawler)
netEstate NE Crawler (+http://www.sengine.info/)
netEstate NE Crawler (+http://www.website-datenbank.de/)
netEstate RSS crawler (+http://www.rss-directory.info/)
nettle (+https://www.nettle.sk)
newspaper/0.0.9.8
newspaper/0.1.0.7
newspaper/0.1.7
newspaper/0.2.5
newspaper/0.2.6
newspaper/0.2.8
nic.it crawler v.1.221115 (7.35.0)
niki-bot
nlcrawler/1.0 (+http://northernlight.com/)
node-fetch/1.0 (+https://github.com/bitinn/node-fetch)
nodestackbot/0.1 (bot@nodestack.com http://nodestack.com/bot.html)
noobot Spider/Noobot-1.2 (noobot-bot; http://www.noobot.fr)
noxtrumbot/1.0 (crawler@noxtrum.com)
nrsbot/5.0(loopip.com/robot.html)
nrsbot/6.0(loopip.com/robot.html)
nuSearch Spider <a href='http://www.nusearch.com'>www.nusearch.com</a> (compatible; MSIE 4.01; Windows NT)
nyobot/1.1 (Noyb.com search engine crawler; http://www.noyb.com; bot@noyb.com),gzip(gfe),gzip(gfe)
oBot
og-scraper/1.0.0
okhttp/2.5.0
okhttp/2.7.5
okhttp/3.2.0
okhttp/3.5.0
okhttp/4.1.0
okhttp/4.9.0
oku-taka-lab-bot (Mozilla-compatible; htakimoto@lr.pi.titech.ac.jp; http://lr-www.pi.titech.ac.jp/wp/)
omgili/0.5 +http://omgili.com
omgilibot/0.3 +http://www.omgili.com/Crawler.html
omgilibot/0.3 http://www.omgili.com/Crawler.html
omgilibot/0.4 +http://omgili.com
online link validator (http://www.dead-links.com/)
ownCloud News/5.3.6 (+https://owncloud.org/; 1 subscriber;)
ownCloud Server Crawler
page_test (larbin2.6.3@unspecified.mail)
page_verifier (http://www.securecomputing.com/goto/pv)
page_viewer (larbin2.6.3@unspecified.mail)
panscient.com
parallelContextFocusCrawler1.1parallelContextFocusCrawler1.1
peerindex/0.1 (http://www.peerindex.com/; crawler AT peerindex DOT com)
percbotspider <ops@percolate.com>
persomm-spider/v1.0
phantomas/1.11.0 (PhantomJS/1.9.8; linux x64)
phantomas/1.8.0 (PhantomJS/1.9.8; linux x64)
php-requests/1.7
phpcrawl
pipeLiner/0.10 (PipeLine Spider; http://www.pipeline-search.com/webmaster.html)
polybot 1.0 (http://cis.poly.edu/polybot/)
pr-cy.ru Screenshot Bot
psbot-image (+http://www.picsearch.com/bot.html)
psbot-page (+http://www.picsearch.com/bot.html)
psbot/0.1 (+http://www.picsearch.com/bot.html)
pyspider/0.3.9 (+http://pyspider.org/)
python-httpx/0.11.1
python-httpx/0.13.0.dev1
python-httpx/0.16.1
python-requests/1.1.0 CPython/2.7.6 Linux/3.13.0-58-generic
python-requests/1.2.0 CPython/2.7.3 Linux/3.2.0-29-virtual
python-requests/2.0.0 CPython/2.6.6 Linux/2.6.32-431.el6.x86_64
python-requests/2.1.0 CPython/2.7.3 Linux/3.2.0-4-amd64
python-requests/2.10.0
python-requests/2.11.1
python-requests/2.18.4
python-requests/2.19.1
python-requests/2.2.1 CPython/2.7.6 Linux/3.13.0-48-generic
python-requests/2.2.1 CPython/2.7.6 Linux/3.13.0-83-generic
python-requests/2.20.0
python-requests/2.21.0
python-requests/2.22.0
python-requests/2.3.0 CPython/2.7.6 Linux/3.13.0-57-generic
python-requests/2.5.0 CPython/2.7.9 Linux/3.14.20-20.44.amzn1.x86_64
python-requests/2.6.0 CPython/2.7.3 Linux/3.2.0-4-amd64
python-requests/2.6.0 CPython/2.7.5 Linux/3.10.0-327.3.1.el7.x86_64
python-requests/2.7.0 CPython/2.7.6 Linux/3.13.0-85-generic
python-requests/2.8.0
python-requests/2.8.1
python-requests/2.9.1
python-requests/2.9.2
pythonic-crawler (suzuki@tkl.iis.u-tokyo.ac.jp)
quickobot/quickobot-1 (Quicko Labs; http://quicko.co; robot at quicko dot co)
radian6_default_(www.radian6.com/crawler)
resolver/0.1.12 (+http://github.com/andris9/resolver)
robots
rogerbot/1.0 (http://moz.com/help/pro/what-is-rogerbot-, rogerbot-crawler+partager@moz.com)
rogerbot/1.0 (http://moz.com/help/pro/what-is-rogerbot-, rogerbot-crawler+shiny@moz.com)
rogerbot/1.0 (http://moz.com/help/pro/what-is-rogerbot-, rogerbot-wherecat@moz.com
rogerbot/1.0 (http://moz.com/help/pro/what-is-rogerbot-, rogerbot-wherecat@moz.com)
rogerbot/1.0 (http://www.moz.com/dp/rogerbot, rogerbot-crawler@moz.com)
rogerbot/1.0 (http://www.seomoz.org/dp/rogerbot, rogerbot-crawler+shiny@seomoz.org)
rogerbot/1.0 (http://www.seomoz.org/dp/rogerbot, rogerbot-crawler@seomoz.org)
rogerbot/1.0 (http://www.seomoz.org/dp/rogerbot, rogerbot-wherecat@moz.com)
rogerbot/1.1 (http://moz.com/help/guides/search-overview/crawl-diagnostics#more-help, rogerbot-crawler+pr2-crawler-05@moz.com)
rogerbot/1.1 (http://moz.com/help/guides/search-overview/crawl-diagnostics#more-help, rogerbot-crawler+pr4-crawler-11@moz.com)
rogerbot/1.1 (http://moz.com/help/guides/search-overview/crawl-diagnostics#more-help, rogerbot-crawler+pr4-crawler-15@moz.com)
rogerbot/1.2 (http://moz.com/help/pro/what-is-rogerbot-, rogerbot-crawler+phaser-testing-crawler-01@moz.com)
rssbot/1.4.3 (+https://t.me/RustRssBot)
ruijie (larbin@unspecified.mail)
sc-downloader/2.0 (http://www.searchconcepts.io)
schibstedsokbot (compatible; Mozilla/5.0; MSIE 5.0; FAST FreshCrawler 6; Contact: webcrawl@schibstedsok.no;)
scrapy-redis (+https://github.com/rolando/scrapy-redis)
scrapyproject (+http://www.profound.net)
seebot/1.0.0 (http://www.seegnify.com/bot)
seebot/2.0 (+http://www.seegnify.com/bot)
seegnifybot/1.0.0 (http://www.seegnify.com/bot)
semanticbot
semanticbot (info@semanticaudience.com)
semanticdiscovery/0.2(http://www.semanticdiscovery.com/sd/robot.html)
semanticdiscovery/2.0(http://www.semanticdiscovery.com/robot.html)
semantics webbot/1.0
sentry/8.18.0 (https://sentry.io)
sentry/8.22.0 (https://sentry.io)
sentry/8.6.0 (https://getsentry.com)
serpstatbot/1.0 (advanced backlink tracking bot; curl/7.58.0; http://serpstatbot.com/; abuse@serpstatbot.com)
serpstatbot/1.0 (advanced backlink tracking bot; http://serpstatbot.com/; abuse@serpstatbot.com)
sg-Orbiter/1.0 (+http://searchgears.de/uber-uns/crawling-faq.html)
shopify-partner-homepage-scraper
shortURL lengthener/0.1 +http://kevindriscoll.info/
silk/1.0 (+http://www.slider.com/silk.htm)/3.7
sixy.ch/1.0
smart.apnoti.com Robot/v1.34 (http://smart.apnoti.com/en/aboutApnotiWebCrawler.html)
snap.com beta crawler v0
socialbm_bot http://spider.socialbm.net
sogou spider
sosoimagespider+(+http://help.soso.com/soso-image-spider.htm)
spider/Nutch-1.5.1 (spider; http://www.xxx.com)
spiderpig/0.1,gzip(gfe),gzip(gfe)
sproose/1.0beta (sproose bot; http://www.sproose.com/bot.html; crawler@sproose.com)
sqlmap/1.0-dev-nongit-201612050a8c (http://sqlmap.org)
sqlmap/1.1.8.2#dev (http://sqlmap.org)
ssearch_bot (sSearch Crawler; http://www.semantissimo.de)
stq_bot (+http://www.searchteq.de)
studylib download bot, see http://studylib.es/public/bot for additional info
support@domainreanimator.com
tCrawler/0.1,gzip(gfe),gzip(gfe),gzip(gfe)
tagSeoBot/1.0 (http://www.tagseoblog.de/tools)
taptubot *** please read http://www.taptu.com/corp/taptubot ***
testnutch/Nutch-1.8
thumbshots-de-bot (+http://www.thumbshots.de/)
tigerbot
tivraSpider/1.0 (crawler@tivra.com)
topicbot/1.0 (Mozilla;I;+http://www.topic.bot/contact_page.html)
trunk.ly spider contact@trunk.ly
truwoGPS/1.0 (GNU/Linux; U; i686; en-US; +http://www.lan4lano.net/browser.html )
uipbot/1.0 (uipbot@semasio.net)
unchaos_crawler_2.0.2 (search.engine@unchaos.com)
unfurlist (https://github.com/Doist/unfurlist)
unshortenit 0.4.0
updated/0.1-alpha (updated crawler; http://www.updated.com; crawler@updated.com)
url_test (larbin2.6.3@unspecified.mail)
urlfan-bot/1.0; +http://www.urlfan.com/site/bot/350.html
vebidoobot
venus/fedoraplanet
veryhip (http://veryhip.com/)
visaduhoc.info Crawler
voyager/2.0 (http://www.kosmix.com/crawler.html)
web_bh (larbin2.6.3@unspecified.mail)
webinatorbot 1.0; +http://www.webinator.de
webinatorbot 1.1; +http://www.webinator.de
webnumbrFetcher/1.0 (http://webnumbr.com/)
weborama-fetcher (+http://www.weborama.com)
webscraper/1.0
websitepulse agent/0.1 (compatible; MSIE 5.5; Netscape 4.75; Linux)
websitepulse checker/1.1 (compatible; MSIE 5.5; Netscape 4.75; Linux)
websitepulse checker/3.0 (compatible; MSIE 5.5; Netscape 4.75; Linux)
wf_crawler (http://websitefigures.com)
wf_crawler (http://www.websitefigures.com)
wikiwix-bot-3.0
wminer/1.2 (North America Web Crawler; http://www.wminer.com/bot; info at wminer dot com)
wonderbot/JS 1.0
woobot
woobot/1.1
woobot/2.0
wp.com feedbot/1.0 (+https://wp.com)
wsAnalyzer/1.0; ++http://www.wsanalyzer.com/bot.html
wscheck.com/1.0.0 (+http://wscheck.com/)
wsr-agent/1.0
wume_crawler/1.1 (http://wume.cse.lehigh.edu/~xiq204/crawler/)
www.deadlinkchecker.com Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/46.0.2490.86 Safari/537.36
www.deadlinkchecker.com XMLHTTP/1.0
www.deadlinkchecker.com XMLHTTP/1.0 Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/46.0.2490.86 Safari/537.36
www.integromedb.org/Crawler
www.website-analyzer.net Website Analyzer Crawler
www2.apserver.net ASP-Ranker Feed Crawler
x28-job-bot; +http://x28.ch/bot.html
yacybot (-global; amd64 FreeBSD 9.2-RELEASE-p10; java 1.7.0_65; Europe/en) http://yacy.net/bot.html
yacybot (-global; amd64 Linux 2.6.32-042stab111.11; java 1.7.0_79; Europe/en) http://yacy.net/bot.html
yacybot (-global; amd64 Linux 2.6.32-042stab116.1; java 1.7.0_79; Europe/en) http://yacy.net/bot.html
yacybot (-global; amd64 Linux 3.10.0-229.4.2.el7.x86_64; java 1.7.0_79; Europe/en) http://yacy.net/bot.html
yacybot (-global; amd64 Linux 3.10.0-229.4.2.el7.x86_64; java 1.8.0_45; Europe/en) http://yacy.net/bot.html
yacybot (-global; amd64 Linux 3.13.0-61-generic; java 1.7.0_79; Europe/en) http://yacy.net/bot.html
yacybot (-global; amd64 Linux 3.14.32-xxxx-grs-ipv6-64; java 1.8.0_111; Europe/de) http://yacy.net/bot.html
yacybot (-global; amd64 Linux 3.16.0-4-amd64; java 1.7.0_75; Europe/en) http://yacy.net/bot.html
yacybot (-global; amd64 Linux 3.19.0-15-generic; java 1.8.0_45-internal; Europe/de) http://yacy.net/bot.html
yacybot (-global; amd64 Linux 3.2.0-4-amd64; java 1.7.0_65; Europe/en) http://yacy.net/bot.html
yacybot (-global; amd64 Linux 3.2.0-4-amd64; java 1.7.0_67; Europe/en) http://yacy.net/bot.html
yacybot (-global; amd64 Linux 4.4.0-57-generic; java 9-internal; Europe/en) http://yacy.net/bot.html
yacybot (-global; amd64 Linux 5.2.11-Jinsol; java 12.0.2; Europe/en) http://yacy.net/bot.html
yacybot (-global; amd64 Linux 5.2.8-Jinsol; java 12.0.2; Europe/en) http://yacy.net/bot.html
yacybot (-global; amd64 Linux 5.2.9-Jinsol; java 12.0.2; Europe/en) http://yacy.net/bot.html
yacybot (-global; amd64 Windows 8 6.2; java 1.7.0_55; Europe/de) http://yacy.net/bot.html
yacybot (-global; amd64 Windows 8.1 6.3; java 1.7.0_55; Europe/de) http://yacy.net/bot.html
yacybot (/global; amd64 FreeBSD 10.3-RELEASE-p7; java 1.7.0_95; GMT/en) http://yacy.net/bot.html
yacybot (/global; amd64 FreeBSD 10.3-RELEASE; java 1.8.0_77; GMT/en) http://yacy.net/bot.html
yacybot (/global; amd64 Linux 2.6.32-042stab093.4; java 1.7.0_65; Etc/en) http://yacy.net/bot.html
yacybot (/global; amd64 Linux 2.6.32-042stab094.8; java 1.7.0_79; America/en) http://yacy.net/bot.html
yacybot (/global; amd64 Linux 2.6.32-042stab108.8; java 1.7.0_91; America/en) http://yacy.net/bot.html
yacybot (/global; amd64 Linux 2.6.32-573.3.1.el6.x86_64; java 1.7.0_85; Europe/en) http://yacy.net/bot.html
yacybot (/global; amd64 Linux 3.10.0-229.7.2.el7.x86_64; java 1.8.0_45; Europe/en) http://yacy.net/bot.html
yacybot (/global; amd64 Linux 3.10.0-327.22.2.el7.x86_64; java 1.7.0_101; Etc/en) http://yacy.net/bot.html
yacybot (/global; amd64 Linux 3.11.10-21-desktop; java 1.7.0_51; America/en) http://yacy.net/bot.html
yacybot (/global; amd64 Linux 3.12.1; java 1.7.0_65; Europe/en) http://yacy.net/bot.html
yacybot (/global; amd64 Linux 3.13.0-042stab093.4; java 1.7.0_79; Europe/de) http://yacy.net/bot.html
yacybot (/global; amd64 Linux 3.13.0-042stab093.4; java 1.7.0_79; Europe/en) http://yacy.net/bot.html
yacybot (/global; amd64 Linux 3.13.0-45-generic; java 1.7.0_75; Europe/en) http://yacy.net/bot.html
yacybot (/global; amd64 Linux 3.13.0-74-generic; java 1.7.0_91; Europe/en) http://yacy.net/bot.html
yacybot (/global; amd64 Linux 3.13.0-83-generic; java 1.7.0_95; Europe/de) http://yacy.net/bot.html
yacybot (/global; amd64 Linux 3.13.0-83-generic; java 1.7.0_95; Europe/en) http://yacy.net/bot.html
yacybot (/global; amd64 Linux 3.13.0-85-generic; java 1.7.0_101; Europe/en) http://yacy.net/bot.html
yacybot (/global; amd64 Linux 3.13.0-85-generic; java 1.7.0_95; Europe/en) http://yacy.net/bot.html
yacybot (/global; amd64 Linux 3.13.0-88-generic; java 1.7.0_101; Europe/en) http://yacy.net/bot.html
yacybot (/global; amd64 Linux 3.14-0.bpo.1-amd64; java 1.7.0_55; Europe/de) http://yacy.net/bot.html
yacybot (/global; amd64 Linux 3.14.32-xxxx-grs-ipv6-64; java 1.7.0_75; Europe/en) http://yacy.net/bot.html
yacybot (/global; amd64 Linux 3.16-0.bpo.2-amd64; java 1.7.0_65; Europe/en) http://yacy.net/bot.html
yacybot (/global; amd64 Linux 3.16.0-4-amd64; java 1.7.0_111; Europe/de) http://yacy.net/bot.html
yacybot (/global; amd64 Linux 3.16.0-4-amd64; java 1.7.0_75; America/en) http://yacy.net/bot.html
yacybot (/global; amd64 Linux 3.16.0-4-amd64; java 1.7.0_75; Europe/en) http://yacy.net/bot.html
yacybot (/global; amd64 Linux 3.16.0-4-amd64; java 1.7.0_79; Europe/de) http://yacy.net/bot.html
yacybot (/global; amd64 Linux 3.16.0-4-amd64; java 1.7.0_79; Europe/en) http://yacy.net/bot.html
yacybot (/global; amd64 Linux 3.16.0-4-amd64; java 1.7.0_91; Europe/de) http://yacy.net/bot.html
yacybot (/global; amd64 Linux 3.16.0-4-amd64; java 1.7.0_95; Europe/en) http://yacy.net/bot.html
yacybot (/global; amd64 Linux 3.16.0-4-amd64; java 1.8.0_102; Europe/en) http://yacy.net/bot.html
yacybot (/global; amd64 Linux 3.16.0-4-amd64; java 1.8.0_111; Europe/en) http://yacy.net/bot.html
yacybot (/global; amd64 Linux 3.16.0-4-amd64; java 1.8.0_40; Europe/en) http://yacy.net/bot.html
yacybot (/global; amd64 Linux 3.16.0-4-amd64; java 1.8.0_91; Europe/en) http://yacy.net/bot.html
yacybot (/global; amd64 Linux 3.16.0-41-generic; java 1.7.0_80; Europe/de) http://yacy.net/bot.html
yacybot (/global; amd64 Linux 3.16.0-49-generic; java 1.7.0_79; Europe/en) http://yacy.net/bot.html
yacybot (/global; amd64 Linux 3.16.0-53-generic; java 1.7.0_85; Europe/en) http://yacy.net/bot.html
yacybot (/global; amd64 Linux 3.16.1-gentoo; java 1.7.0_55; Europe/en) http://yacy.net/bot.html
yacybot (/global; amd64 Linux 3.19.0-26-generic; java 1.7.0_95; Europe/en) http://yacy.net/bot.html
yacybot (/global; amd64 Linux 3.19.2-1-ARCH; java 1.8.0_40; Europe/de) http://yacy.net/bot.html
yacybot (/global; amd64 Linux 3.2.0-4-amd64; java 1.7.0_03; Europe/en) http://yacy.net/bot.html
yacybot (/global; amd64 Linux 3.2.0-4-amd64; java 1.7.0_101; Europe/cs) http://yacy.net/bot.html
yacybot (/global; amd64 Linux 3.2.0-4-amd64; java 1.7.0_60; Europe/en) http://yacy.net/bot.html
yacybot (/global; amd64 Linux 3.2.0-4-amd64; java 1.7.0_65; Europe/en) http://yacy.net/bot.html
yacybot (/global; amd64 Linux 3.2.0-4-amd64; java 1.7.0_75; Europe/en) http://yacy.net/bot.html
yacybot (/global; amd64 Linux 3.2.0-4-amd64; java 1.7.0_79; Europe/en) http://yacy.net/bot.html
yacybot (/global; amd64 Linux 3.2.0-4-amd64; java 1.8.0_05; Europe/en) http://yacy.net/bot.html
yacybot (/global; amd64 Linux 3.2.41-042stab094.7; java 1.7.0_75; Europe/en) http://yacy.net/bot.html
yacybot (/global; amd64 Linux 3.2.41-042stab094.7; java 1.7.0_79; Europe/en) http://yacy.net/bot.html
yacybot (/global; amd64 Linux 4.0.5-gentoo; java 1.8.0_51; Europe/en) http://yacy.net/bot.html
yacybot (/global; amd64 Linux 4.0.7-1-ck; java 1.8.0_45; Europe/de) http://yacy.net/bot.html
yacybot (/global; amd64 Linux 4.0.9-2-ck; java 1.8.0_51; Europe/de) http://yacy.net/bot.html
yacybot (/global; amd64 Linux 4.1.19-gentoo; java 1.7.0_95; Europe/pl) http://yacy.net/bot.html
yacybot (/global; amd64 Linux 4.2.0-22-generic; java 1.7.0_91; Europe/en) http://yacy.net/bot.html
yacybot (/global; amd64 Linux 4.2.0-27-generic; java 1.7.0_95; Europe/en) http://yacy.net/bot.html
yacybot (/global; amd64 Linux 4.2.0-27-generic; java 1.8.0_66-internal; America/en) http://yacy.net/bot.html
yacybot (/global; amd64 Linux 4.2.0-gentoo-r1; java 1.8.0_45; Australia/en) http://yacy.net/bot.html
yacybot (/global; amd64 Linux 4.3.0-gentoo-ARCH; java 1.7.0_85; Europe/en) http://yacy.net/bot.html
yacybot (/global; amd64 Linux 4.4.0-28-generic; java 1.8.0_91; Europe/pl) http://yacy.net/bot.html
yacybot (/global; amd64 Linux 4.4.0-31-generic; java 1.8.0_91; Europe/en) http://yacy.net/bot.html
yacybot (/global; amd64 Linux 4.4.0-59-generic; java 1.8.0_121; Europe/de) http://yacy.net/bot.html
yacybot (/global; amd64 Linux 4.4.0ct; java 1.7.0_91; Europe/de) http://yacy.net/bot.html
yacybot (/global; amd64 Linux 4.4.10-antix.1-amd64-smp; java 1.8.0_101; Europe/en) http://yacy.net/bot.html
yacybot (/global; amd64 Linux 4.4.5-1-ARCH; java 1.8.0_77; America/en) http://yacy.net/bot.html
yacybot (/global; amd64 Linux 4.7.6-200.fc24.x86_64; java 1.8.0_102; Etc/en) http://yacy.net/bot.html
yacybot (/global; amd64 Windows 7 6.1; java 1.7.0_55; Asia/zh) http://yacy.net/bot.html
yacybot (/global; amd64 Windows 7 6.1; java 1.7.0_55; Europe/de) http://yacy.net/bot.html
yacybot (/global; amd64 Windows 7 6.1; java 1.7.0_55; Europe/en) http://yacy.net/bot.html
yacybot (/global; amd64 Windows 7 6.1; java 1.7.0_55; Europe/fr) http://yacy.net/bot.html
yacybot (/global; amd64 Windows 7 6.1; java 1.7.0_55; Europe/ru) http://yacy.net/bot.html
yacybot (/global; amd64 Windows 7 6.1; java 1.7.0_67; Europe/de) http://yacy.net/bot.html
yacybot (/global; amd64 Windows 7 6.1; java 1.7.0_72; Europe/de) http://yacy.net/bot.html
yacybot (/global; amd64 Windows 7 6.1; java 1.8.0_05; Europe/es) http://yacy.net/bot.html
yacybot (/global; amd64 Windows 7 6.1; java 1.8.0_101; Asia/ru) http://yacy.net/bot.html
yacybot (/global; amd64 Windows 7 6.1; java 1.8.0_25; Europe/en) http://yacy.net/bot.html
yacybot (/global; amd64 Windows 7 6.1; java 1.8.0_45; Europe/de) http://yacy.net/bot.html
yacybot (/global; amd64 Windows 7 6.1; java 1.8.0_51; Europe/de) http://yacy.net/bot.html
yacybot (/global; amd64 Windows 7 6.1; java 1.8.0_65; Europe/de) http://yacy.net/bot.html
yacybot (/global; amd64 Windows 7 6.1; java 1.8.0_66; Europe/de) http://yacy.net/bot.html
yacybot (/global; amd64 Windows 8.1 6.3; java 1.7.0_55; America/ru) http://yacy.net/bot.html
yacybot (/global; amd64 Windows 8.1 6.3; java 1.7.0_55; Europe/de) http://yacy.net/bot.html
yacybot (/global; amd64 Windows 8.1 6.3; java 1.8.0_111; Europe/de) http://yacy.net/bot.html
yacybot (/global; amd64 Windows 8.1 6.3; java 1.8.0_25; Europe/de) http://yacy.net/bot.html
yacybot (/global; amd64 Windows 8.1 6.3; java 1.8.0_40; Europe/de) http://yacy.net/bot.html
yacybot (/global; amd64 Windows 8.1 6.3; java 1.8.0_45; Europe/ru) http://yacy.net/bot.html
yacybot (/global; amd64 Windows Server 2008 R2 6.1; java 1.8.0_40; Europe/de) http://yacy.net/bot.html
yacybot (/global; amd64 Windows Server 2008 R2 6.1; java 1.8.0_60; Europe/de) http://yacy.net/bot.html
yacybot (/global; amd64 Windows Server 2012 6.2; java 1.7.0_51; Europe/de) http://yacy.net/bot.html
yacybot (/global; amd64 Windows Server 2012 6.2; java 1.8.0_91; Europe/de) http://yacy.net/bot.html
yacybot (/global; amd64 Windows Server 2012 R2 6.3; java 1.7.0_55; Europe/de) http://yacy.net/bot.html
yacybot (/global; amd64 Windows Vista 6.0; java 1.7.0_55; America/en) http://yacy.net/bot.html
yacybot (/global; amd64 no-os-name no-os-version; java no-java-version; Europe/en) http://yacy.net/bot.html
yacybot (/global; arm Linux 3.12.28+; java 1.7.0_71; Europe/en) http://yacy.net/bot.html
yacybot (/global; arm Linux 4.1.13+; java 1.8.0_40-internal; Etc/de) http://yacy.net/bot.html
yacybot (/global; arm Linux 4.1.13-v7+; java 1.8.0; Etc/de) http://yacy.net/bot.html
yacybot (/global; i386 Linux 3.13.0-37-generic; java 1.7.0_65; Europe/en) http://yacy.net/bot.html
yacybot (/global; i386 Linux 3.16.3-gentoo; java 1.7.0_55; Australia/en) http://yacy.net/bot.html
yacybot (/global; x86 Windows 10 10.0; java 1.8.0_73; Europe/de) http://yacy.net/bot.html
yacybot (/global; x86 Windows 2003 5.2; java 1.7.0_67; Europe/de) http://yacy.net/bot.html
yacybot (/global; x86 Windows 7 6.1; java 1.8.0_31; Europe/de) http://yacy.net/bot.html
yacybot (/global; x86 Windows 7 6.1; java 1.8.0_40; America/es) http://yacy.net/bot.html
yacybot (/global; x86 Windows 7 6.1; java 1.8.0_71; Europe/de) http://yacy.net/bot.html
yacybot (/global; x86 Windows 8.1 6.3; java 1.8.0_45; America/en) http://yacy.net/bot.html
yacybot (/global; x86 Windows XP 5.1; java 1.7.0_51; Europe/de) http://yacy.net/bot.html
yacybot (/global; x86 Windows XP 5.1; java 1.7.0_55; Europe/de) http://yacy.net/bot.html
yacybot (/global; x86 Windows XP 5.1; java 1.7.0_71; Europe/de) http://yacy.net/bot.html
yacybot (amd64 Linux 2.6.18-164.15.1.el5xen; java 1.6.0_0; Europe/en) http://yacy.net/bot.html
yacybot (amd64 Linux 2.6.18-164.el5; java 1.6.0; Europe/de) http://yacy.net/bot.html
yacybot (amd64 Linux 2.6.24-23-generic; java 1.6.0_07; Europe/de) http://yacy.net/bot.html
yacybot (amd64 Linux 2.6.26-2-amd64; java 1.6.0_0; Europe/en) http://yacy.net/bot.html
yacybot (amd64 Linux 2.6.26-2-amd64; java 1.6.0_20; Europe/en) http://yacy.net/bot.html
yacybot (amd64 Linux 2.6.26-2-openvz-amd64; java 1.6.0_12; UTC/en) http://yacy.net/bot.html
yacybot (amd64 Linux 2.6.28-18-generic; java 1.6.0_16; GMT/en) http://yacy.net/bot.html
yacybot (amd64 Linux 2.6.31-22-server; java 1.6.0_18; Asia/en) http://yacy.net/bot.html
yacybot (amd64 Linux 2.6.32-gentoo; java 1.6.0_17; Europe/en) http://yacy.net/bot.html
yacybot (amd64 Windows 7 6.1; java 1.6.0_14; Europe/de) http://yacy.net/bot.html
yacybot (amd64 Windows 7 6.1; java 1.6.0_18; Europe/de) http://yacy.net/bot.html
yacybot (amd64 Windows 7 6.1; java 1.6.0_21; Europe/fr) http://yacy.net/bot.html
yacybot (freeworld-global; amd64 Linux 3.0.0-21-generic; java 1.6.0_26; America/en) http://yacy.net/bot.html
yacybot (freeworld-global; amd64 Linux 3.16.0-4-amd64; java 1.7.0_79; Europe/de) http://yacy.net/bot.html
yacybot (freeworld-global; amd64 Linux 3.2.0-35-generic; java 1.7.0_09; Europe/en) http://yacy.net/bot.html
yacybot (freeworld-global; amd64 Linux 3.2.0-4-amd64; java 1.6.0_24; Europe/en) http://yacy.net/bot.html
yacybot (freeworld-global; amd64 Windows 7 6.1; java 1.7.0_02-ea; America/en) http://yacy.net/bot.html
yacybot (freeworld/global; amd64 Linux 2.6.26-2-amd64; java 1.6.0_18; Europe/de) http://yacy.net/bot.html
yacybot (freeworld/global; amd64 Linux 2.6.32-34-server; java 1.6.0_26; Europe/de) http://yacy.net/bot.html
yacybot (freeworld/global; amd64 Linux 2.6.32-40-generic; java 1.6.0_20; Europe/de) http://yacy.net/bot.html
yacybot (freeworld/global; amd64 Linux 2.6.32-40-server; java 1.6.0_20; Europe/en) http://yacy.net/bot.html
yacybot (freeworld/global; amd64 Linux 2.6.32-41-server; java 1.6.0_26; Europe/de) http://yacy.net/bot.html
yacybot (freeworld/global; amd64 Linux 2.6.32-431.29.2.el6.x86_64; java 1.7.0_65; Europe/en) http://yacy.net/bot.html
yacybot (freeworld/global; amd64 Linux 2.6.32-49-server; java 1.6.0_27; Europe/en) http://yacy.net/bot.html
yacybot (freeworld/global; amd64 Linux 2.6.32-5-amd64; java 1.6.0_18; Europe/de) http://yacy.net/bot.html
yacybot (freeworld/global; amd64 Linux 2.6.32-5-amd64; java 1.6.0_18; Europe/en) http://yacy.net/bot.html
yacybot (freeworld/global; amd64 Linux 2.6.32-5-amd64; java 1.6.0_18; US/en) http://yacy.net/bot.html
yacybot (freeworld/global; amd64 Linux 2.6.32-5-amd64; java 1.6.0_26; Atlantic/en) http://yacy.net/bot.html
yacybot (freeworld/global; amd64 Linux 2.6.32-5-xen-amd64; java 1.6.0_18; Europe/fr) http://yacy.net/bot.html
yacybot (freeworld/global; amd64 Linux 2.6.32-custom; java 1.6.0_26; Europe/en) http://yacy.net/bot.html
yacybot (freeworld/global; amd64 Linux 2.6.38-14-generic; java 1.6.0_22; Europe/en) http://yacy.net/bot.html
yacybot (freeworld/global; amd64 Linux 2.6.38-8-generic; java 1.6.0_22; Europe/en) http://yacy.net/bot.html
yacybot (freeworld/global; amd64 Linux 3.0.0-12-generic; java 1.6.0_26; Europe/en) http://yacy.net/bot.html
yacybot (freeworld/global; amd64 Linux 3.0.0-14-generic; java 1.6.0_23; Europe/en) http://yacy.net/bot.html
yacybot (freeworld/global; amd64 Linux 3.0.0-15-server; java 1.6.0_23; Europe/en) http://yacy.net/bot.html
yacybot (freeworld/global; amd64 Linux 3.0.0-17-generic; java 1.6.0_23; America/en) http://yacy.net/bot.html
yacybot (freeworld/global; amd64 Linux 3.0.0-17-generic; java 1.6.0_23; Europe/de) http://yacy.net/bot.html
yacybot (freeworld/global; amd64 Linux 3.0.0-21-generic; java 1.6.0_26; America/en) http://yacy.net/bot.html
yacybot (freeworld/global; amd64 Linux 3.1.10-1-desktop; java 1.6.0_22; Europe/de) http://yacy.net/bot.html
yacybot (freeworld/global; amd64 Linux 3.1.10-1.9-default; java 1.6.0_24; Europe/de) http://yacy.net/bot.html
yacybot (freeworld/global; amd64 Linux 3.1.10-hardened; java 1.7.0_03-icedtea; Europe/en) http://yacy.net/bot.html
yacybot (freeworld/global; amd64 Linux 3.10.15-1-MANJARO; java 1.7.0_40; Asia/en) http://yacy.net/bot.html
yacybot (freeworld/global; amd64 Linux 3.10.17-gentoo; java 1.7.0_45; UTC/en) http://yacy.net/bot.html
yacybot (freeworld/global; amd64 Linux 3.10.17-gentooxen; java 1.6.0_31; Europe/nb) http://yacy.net/bot.html
yacybot (freeworld/global; amd64 Linux 3.11.10-21-desktop; java 1.7.0_51; Europe/de) http://yacy.net/bot.html
yacybot (freeworld/global; amd64 Linux 3.12.15-gentoo; java 1.7.0_55; Europe/en) http://yacy.net/bot.html
yacybot (freeworld/global; amd64 Linux 3.12.43-52.6-default; java 1.8.0_40; Europe/en) http://yacy.net/bot.html
yacybot (freeworld/global; amd64 Linux 3.13.0-24-generic; java 1.7.0_55; Europe/de) http://yacy.net/bot.html
yacybot (freeworld/global; amd64 Linux 3.2.0-2-amd64; java 1.6.0_24; Europe/en) http://yacy.net/bot.html
yacybot (freeworld/global; amd64 Linux 3.2.0-21-generic; java 1.7.0_03-icedtea; America/en) http://yacy.net/bot.html
yacybot (freeworld/global; amd64 Linux 3.2.0-23-generic; java 1.6.0_24; Europe/en) http://yacy.net/bot.html
yacybot (freeworld/global; amd64 Linux 3.2.0-24-generic; java 1.6.0_24; Europe/en) http://yacy.net/bot.html
yacybot (freeworld/global; amd64 Linux 3.2.0-3-amd64; java 1.6.0_24; Europe/en) http://yacy.net/bot.html
yacybot (freeworld/global; amd64 Linux 3.2.0-32-generic; java 1.6.0_24; Europe/en) http://yacy.net/bot.html
yacybot (freeworld/global; amd64 Linux 3.2.0-4-amd64; java 1.6.0_27; Europe/en) http://yacy.net/bot.html
yacybot (freeworld/global; amd64 Linux 3.2.0-4-amd64; java 1.7.0_03; Etc/de) http://yacy.net/bot.html
yacybot (freeworld/global; amd64 Linux 3.2.0-4-amd64; java 1.7.0_25; Europe/de) http://yacy.net/bot.html
yacybot (freeworld/global; amd64 Linux 3.2.0-4-amd64; java 1.7.0_25; Europe/en) http://yacy.net/bot.html
yacybot (freeworld/global; amd64 Linux 3.2.0-4-amd64; java 1.7.0_55; Europe/de) http://yacy.net/bot.html
yacybot (freeworld/global; amd64 Linux 3.2.1-gentoo-r2; java 1.6.0_22; Europe/de) http://yacy.net/bot.html
yacybot (freeworld/global; amd64 Linux 3.2.1-gentoo-r2; java 1.6.0_24; Europe/de) http://yacy.net/bot.html
yacybot (freeworld/global; amd64 Linux 3.2.13-1-ARCH; java 1.7.0_03-icedtea; Europe/fr) http://yacy.net/bot.html
yacybot (freeworld/global; amd64 Linux 3.3.4-1-ARCH; java 1.6.0_24; Europe/en) http://yacy.net/bot.html
yacybot (freeworld/global; amd64 Linux 3.5.0-27-generic; java 1.7.0_03; Europe/de) http://yacy.net/bot.html
yacybot (freeworld/global; amd64 Linux 3.5.0-27-generic; java 1.7.0_25; Europe/en) http://yacy.net/bot.html
yacybot (freeworld/global; amd64 Linux 3.8.0-19-generic; java 1.7.0_25; Europe/en) http://yacy.net/bot.html
yacybot (freeworld/global; amd64 Linux 3.8.0-21-generic; java 1.6.0_27; Pacific/en) http://yacy.net/bot.html
yacybot (freeworld/global; amd64 Linux 3.8.0-23-generic; java 1.6.0_27; Pacific/en) http://yacy.net/bot.html
yacybot (freeworld/global; amd64 Linux 3.8.13-gentoo; java 1.7.0_21; Europe/de) http://yacy.net/bot.html
yacybot (freeworld/global; amd64 Linux 3.9.3-1-ARCH; java 1.7.0_21; America/en) http://yacy.net/bot.html
yacybot (freeworld/global; amd64 Windows 10 10.0; java 1.8.0_60; Europe/de) http://yacy.net/bot.html
yacybot (freeworld/global; amd64 Windows 7 6.1; java 1.6.0_23; Europe/de) http://yacy.net/bot.html
yacybot (freeworld/global; amd64 Windows 7 6.1; java 1.6.0_24; Europe/de) http://yacy.net/bot.html
yacybot (freeworld/global; amd64 Windows 7 6.1; java 1.6.0_25; Europe/de) http://yacy.net/bot.html
yacybot (freeworld/global; amd64 Windows 7 6.1; java 1.6.0_29; Europe/fr) http://yacy.net/bot.html
yacybot (freeworld/global; amd64 Windows 7 6.1; java 1.6.0_31; America/en) http://yacy.net/bot.html
yacybot (freeworld/global; amd64 Windows 7 6.1; java 1.6.0_31; Europe/de) http://yacy.net/bot.html
yacybot (freeworld/global; amd64 Windows 7 6.1; java 1.7.0_04; Asia/ja) http://yacy.net/bot.html
yacybot (freeworld/global; amd64 Windows 7 6.1; java 1.7.0_09; Europe/de) http://yacy.net/bot.html
yacybot (freeworld/global; amd64 Windows 7 6.1; java 1.7.0_25; Europe/de) http://yacy.net/bot.html
yacybot (freeworld/global; amd64 Windows 7 6.1; java 1.7.0_45; Europe/de) http://yacy.net/bot.html
yacybot (freeworld/global; amd64 Windows 7 6.1; java 1.7.0_60; Europe/de) http://yacy.net/bot.html
yacybot (freeworld/global; amd64 Windows 7 6.1; java 1.8.0_11; Europe/de) http://yacy.net/bot.html
yacybot (freeworld/global; amd64 Windows 8 6.2; java 1.7.0_25; Europe/de) http://yacy.net/bot.html
yacybot (freeworld/global; amd64 Windows 8 6.2; java 1.7.0_51; Europe/de) http://yacy.net/bot.html
yacybot (freeworld/global; amd64 Windows NT (unknown) 6.2; java 1.7.0_05; Africa/en) http://yacy.net/bot.html
yacybot (freeworld/global; amd64 Windows Server 2008 6.0; java 1.7.0_03; Europe/en) http://yacy.net/bot.html
yacybot (freeworld/global; amd64 Windows Server 2008 6.0; java 1.7.0_25; Europe/en) http://yacy.net/bot.html
yacybot (freeworld/global; amd64 Windows Server 2008 R2 6.1; java 1.6.0_29; Europe/en) http://yacy.net/bot.html
yacybot (freeworld/global; amd64 Windows Server 2008 R2 6.1; java 1.6.0_31; America/pt) http://yacy.net/bot.html
yacybot (freeworld/global; amd64 Windows Server 2008 R2 6.1; java 1.7.0_25; Europe/de) http://yacy.net/bot.html
yacybot (freeworld/global; amd64 Windows Server 2012 6.2; java 1.7.0_25; Europe/de) http://yacy.net/bot.html
yacybot (freeworld/global; amd64 Windows Server 2012 6.2; java 1.7.0_51; Europe/de) http://yacy.net/bot.html
yacybot (freeworld/global; amd64 Windows XP 5.2; java 1.7.0_04; America/en) http://yacy.net/bot.html
yacybot (freeworld/global; arm Linux 4.4.11-v7+; java 1.7.0_101; Etc/en) http://yacy.net/bot.html
yacybot (freeworld/global; i386 Linux 2.6.32-39-generic-pae; java 1.6.0_20; Europe/en) http://yacy.net/bot.html
yacybot (freeworld/global; i386 Linux 2.6.32-5-686; java 1.6.0_18; Europe/en) http://yacy.net/bot.html
yacybot (freeworld/global; i386 Linux 2.6.37.6-0.5-desktop; java 1.6.0_20; Europe/en) http://yacy.net/bot.html
yacybot (freeworld/global; i386 Linux 3.0.0-17-generic-pae; java 1.6.0_23; Europe/en) http://yacy.net/bot.html
yacybot (freeworld/global; i386 Linux 3.0.0-17-generic; java 1.6.0_23; America/en) http://yacy.net/bot.html
yacybot (freeworld/global; i386 Linux 3.12-1-686-pae; java 1.7.0_21; Europe/fr) http://yacy.net/bot.html
yacybot (freeworld/global; i386 Linux 3.2.0-23-generic-pae; java 1.7.0_03; Europe/en) http://yacy.net/bot.html
yacybot (freeworld/global; i386 Linux 3.2.0-23-generic; java 1.6.0_27; Europe/en) http://yacy.net/bot.html
yacybot (freeworld/global; i386 Linux 3.4.2-linode44; java 1.6.0_27; America/en) http://yacy.net/bot.html
yacybot (freeworld/global; x86 Windows 2003 5.2; java 1.6.0_32; Europe/de) http://yacy.net/bot.html
yacybot (freeworld/global; x86 Windows 7 6.1; java 1.6.0_31; Europe/de) http://yacy.net/bot.html
yacybot (freeworld/global; x86 Windows 7 6.1; java 1.7.0_25; Europe/de) http://yacy.net/bot.html
yacybot (freeworld/global; x86 Windows 7 6.1; java 1.8.0_25; Europe/de) http://yacy.net/bot.html
yacybot (freeworld/global; x86 Windows XP 5.1; java 1.7.0_21; GMT+04:00/ru) http://yacy.net/bot.html
yacybot (freeworld/global; x86 Windows XP 5.1; java 1.7.0_55; Asia/en) http://yacy.net/bot.html
yacybot (freeworld/global; x86_64 Mac OS X 10.6.8; java 1.6.0_29; Asia/ru) http://yacy.net/bot.html
yacybot (i386 Linux 2.6.23; java 1.6.0_06; Europe/en) http://yacy.net/bot.html
yacybot (i386 Linux 2.6.23; java 1.6.0_17; Europe/en) http://yacy.net/bot.html
yacybot (i386 Linux 2.6.24-23-generic; java 1.6.0_16; Europe/en) http://yacy.net/bot.html
yacybot (i386 Linux 2.6.26-2-686; java 1.6.0_0; Europe/de) http://yacy.net/bot.html
yacybot (i386 Linux 2.6.28-11-generic; java 1.6.0_13; Europe/de) http://yacy.net/bot.html
yacybot (i386 Linux 2.6.28-13-generic; java 1.6.0_13; Europe/en) http://yacy.net/bot.html
yacybot (i386 Linux 2.6.28-gentoo-r5; java 1.5.0_18; Europe/en) http://yacy.net/bot.html
yacybot (i386 Linux 2.6.30-2-686; java 1.6.0_0; SystemV/en) http://yacy.net/bot.html
yacybot (i386 Linux 2.6.31-16-generic; java 1.6.0_15; Europe/en) http://yacy.net/bot.html
yacybot (i386 Linux 2.6.31-18-generic; java 1.6.0_0; Europe/en) http://yacy.net/bot.html
yacybot (i386 Linux 2.6.31-21-generic; java 1.6.0_0; Europe/en) http://yacy.net/bot.html
yacybot (i386 Linux 2.6.32-22-generic; java 1.6.0_20; Europe/de) http://yacy.net/bot.html
yacybot (i386 Linux 2.6.9-023stab046.2-smp; java 1.6.0_05; Europe/en) http://yacy.net/bot.html
yacybot (ppc Mac OS X 10.5.2; java 1.5.0_13; Europe/de) http://yacy.net/bot.html
yacybot (webportal-global; amd64 Linux 2.6.32-5-amd64; java 1.6.0_18; Europe/en) http://yacy.net/bot.html
yacybot (webportal-global; amd64 Linux 2.6.32-5-amd64; java 1.6.0_45; Europe/en) http://yacy.net/bot.html
yacybot (webportal-global; amd64 Linux 3.2.0-4-amd64; java 1.7.0_25; Europe/en) http://yacy.net/bot.html
yacybot (webportal-global; amd64 Linux 3.2.0-4-amd64; java 1.7.0_65; Europe/en) http://yacy.net/bot.html
yacybot (webportal-global; amd64 Linux 3.2.0-4-amd64; java 1.7.0_67; Europe/en) http://yacy.net/bot.html
yacybot (webportal-global; amd64 Linux 4.4.0-46-generic; java 1.8.0_91; Europe/en) http://yacy.net/bot.html
yacybot (webportal-global; amd64 Windows 7 6.1; java 1.7.0_04; America/en) http://yacy.net/bot.html
yacybot (webportal-global; x86 Windows Vista 6.0; java 1.7.0_25; Europe/en) http://yacy.net/bot.html
yacybot (webportal/global; amd64 Linux 2.6.32-5-amd64; java 1.6.0_18; Europe/en) http://yacy.net/bot.html
yacybot (webportal/global; x86_64 Mac OS X 10.9.2; java 1.6.0_65; Europe/de) http://yacy.net/bot.html
yacybot (x86 Windows 2003 5.2; java 1.6.0_16; Europe/de) http://yacy.net/bot.html
yacybot (x86 Windows 2003 5.2; java 1.6.0_20; America/de) http://yacy.net/bot.html
yacybot (x86 Windows Vista 6.1; java 1.6.0_13; Europe/de) http://yacy.net/bot.html
yacybot (x86 Windows XP 5.1; java 1.6.0_18; Europe/de) http://yacy.net/bot.html
yacybot (x86 Windows XP 5.1; java 1.6.0_21; Europe/de) http://yacy.net/bot.html
your sipder name/Nutch-1.7
yrspider (Mozilla/5.0 (compatible; YRSpider; +http://www.yunrang.com/yrspider.html))
yrspider (Mozilla/5.0 (compatible; YRSpider; http://www.yunrang.com/yrspider.html))
zelist.ro feed parser (+http://www.zelist.ro)
zspider/0.9-dev http://feedback.redkolibri.com/
Goose/3.1.11