User-agent: * Allow: / User-agent: Mediapartners-Google Allow: / User-agent: Googlebot Allow: / User-agent: Adsbot-Google Allow: / User-agent: Googlebot-Image Allow: / User-agent: Googlebot-Mobile Allow: / User-agent: Slurp Allow: / User-agent: msnbot Allow: / User-agent: Teoma Allow: / User-agent: msnbot Allow: / User-agent: AdSense program Allow: / User-agent: Baiduspider Allow: / User-agent: ABCdatos BotLink Allow: / User-agent: Acme.Spider Allow: / User-agent: Ahoy! The Homepage Finder Allow: / User-agent: Alkaline Allow: / User-agent: Anthill Allow: / User-agent: Walhello appie Allow: / User-agent: Arachnophilia Allow: / User-agent: Arale Allow: / User-agent: Araneo Allow: / User-agent: AraybOt Allow: / User-agent: ArchitextSpider Allow: / User-agent: Aretha Allow: / User-agent: ARIADNE Allow: / User-agent: arks Allow: / User-agent: AskJeeves Allow: / User-agent: ASpider (Associative Spider) Allow: / User-agent: ATN Worldwide Allow: / User-agent: Atomz.com Search Robot Allow: / User-agent: AURESYS Allow: / User-agent: BackRub Allow: / User-agent: Bay Spider Allow: / User-agent: Big Brother Allow: / User-agent: Bjaaland Allow: / User-agent: BlackWidow Allow: / User-agent: Die Blinde Kuh Allow: / User-agent: Bloodhound Allow: / User-agent: Borg-Bot Allow: / User-agent: BoxSeaBot Allow: / User-agent: bright.net caching robot Allow: / User-agent: BSpider Allow: / User-agent: CACTVS Chemistry Spider Allow: / User-agent: Calif Allow: / User-agent: Cassandra Allow: / User-agent: Digimarc Marcspider/CGI Allow: / User-agent: Checkbot Allow: / User-agent: ChristCrawler.com Allow: / User-agent: churl Allow: / User-agent: cIeNcIaFiCcIoN.nEt Allow: / User-agent: CMC/0.01 Allow: / User-agent: Collective Allow: / User-agent: Combine System Allow: / User-agent: Conceptbot Allow: / User-agent: ConfuzzledBot Allow: / User-agent: CoolBot Allow: / User-agent: Web Core / Roots Allow: / User-agent: XYLEME Robot Allow: / User-agent: Internet Cruiser Robot Allow: / User-agent: Cusco Allow: / User-agent: CyberSpyder Link Test Allow: / User-agent: CydralSpider Allow: / User-agent: Desert Realm Spider Allow: / User-agent: DeWeb(c) Katalog/Index Allow: / User-agent: DienstSpider Allow: / User-agent: Digger Allow: / User-agent: Digital Integrity Robot Allow: / User-agent: Direct Hit Grabber Allow: / User-agent: DNAbot Allow: / User-agent: DownLoad Express Allow: / User-agent: DragonBot Allow: / User-agent: DWCP (Dridus' Web Cataloging Project) Allow: / User-agent: e-collector Allow: / User-agent: EbiNess Allow: / User-agent: EIT Link Verifier Robot Allow: / User-agent: ELFINBOT Allow: / User-agent: Emacs-w3 Search Engine Allow: / User-agent: ananzi Allow: / User-agent: esculapio Allow: / User-agent: Esther Allow: / User-agent: Evliya Celebi Allow: / User-agent: FastCrawler Allow: / User-agent: Fluid Dynamics Search Engine robot Allow: / User-agent: Felix IDE Allow: / User-agent: Wild Ferret Web Hopper #1, #2, #3 Allow: / User-agent: FetchRover Allow: / User-agent: fido Allow: / User-agent: Hämähäkki Allow: / User-agent: KIT-Fireball Allow: / User-agent: Fish search Allow: / User-agent: Fouineur Allow: / User-agent: Robot Francoroute Allow: / User-agent: Freecrawl Allow: / User-agent: FunnelWeb Allow: / User-agent: gammaSpider, FocusedCrawler Allow: / User-agent: gazz Allow: / User-agent: GCreep Allow: / User-agent: GetBot Allow: / User-agent: GetURL Allow: / User-agent: Golem Allow: / User-agent: Googlebot Allow: / User-agent: Grapnel/0.01 Experiment Allow: / User-agent: Griffon Allow: / User-agent: Gromit Allow: / User-agent: Northern Light Gulliver Allow: / User-agent: Gulper Bot Allow: / User-agent: HamBot Allow: / User-agent: Harvest Allow: / User-agent: havIndex Allow: / User-agent: HI (HTML Index) Search Allow: / User-agent: Hometown Spider Pro Allow: / User-agent: ht: //Dig Allow: / User-agent: HTMLgobble Allow: / User-agent: Hyper-Decontextualizer Allow: / User-agent: iajaBot Allow: / User-agent: IBM_Planetwide Allow: / User-agent: Popular Iconoclast Allow: / User-agent: Ingrid Allow: / User-agent: Imagelock Allow: / User-agent: IncyWincy Allow: / User-agent: Informant Allow: / User-agent: InfoSeek Robot 1.0 Allow: / User-agent: Infoseek Sidewinder Allow: / User-agent: InfoSpiders Allow: / User-agent: Inspector Web Allow: / User-agent: IntelliAgent Allow: / User-agent: I, Robot Allow: / User-agent: Iron33 Allow: / User-agent: Israeli-search Allow: / User-agent: JavaBee Allow: / User-agent: JBot Java Web Robot Allow: / User-agent: JCrawler Allow: / User-agent: Jeeves Allow: / User-agent: JoBo Java Web Robot Allow: / User-agent: Jobot Allow: / User-agent: JoeBot Allow: / User-agent: The Jubii Indexing Robot Allow: / User-agent: JumpStation Allow: / User-agent: image.kapsi.net Allow: / User-agent: Katipo Allow: / User-agent: KDD-Explorer Allow: / User-agent: Kilroy Allow: / User-agent: KO_Yappo_Robot Allow: / User-agent: LabelGrabber Allow: / User-agent: larbin Allow: / User-agent: legs Allow: / User-agent: Link Validator Allow: / User-agent: LinkScan Allow: / User-agent: LinkWalker Allow: / User-agent: Lockon Allow: / User-agent: logo.gif Crawler Allow: / User-agent: Lycos Allow: / User-agent: Mac WWWWorm Allow: / User-agent: Magpie Allow: / User-agent: marvin/infoseek Allow: / User-agent: Mattie Allow: / User-agent: MediaFox Allow: / User-agent: MerzScope Allow: / User-agent: NEC-MeshExplorer Allow: / User-agent: MindCrawler Allow: / User-agent: mnoGoSearch search engine software Allow: / User-agent: moget Allow: / User-agent: MOMspider Allow: / User-agent: Monster Allow: / User-agent: Motor Allow: / User-agent: MSNBot Allow: / User-agent: Muncher Allow: / User-agent: Muninn Allow: / User-agent: Muscat Ferret Allow: / User-agent: Mwd.Search Allow: / User-agent: Internet Shinchakubin Allow: / User-agent: NDSpider Allow: / User-agent: Nederland.zoek Allow: / User-agent: NetCarta WebMap Engine Allow: / User-agent: NetMechanic Allow: / User-agent: NetScoop Allow: / User-agent: newscan-online Allow: / User-agent: NHSE Web Forager Allow: / User-agent: Nomad Allow: / User-agent: The NorthStar Robot Allow: / User-agent: nzexplorer Allow: / User-agent: ObjectsSearch Allow: / User-agent: Occam Allow: / User-agent: HKU WWW Octopus Allow: / User-agent: OntoSpider Allow: / User-agent: Openfind data gatherer Allow: / User-agent: Orb Search Allow: / User-agent: Pack Rat Allow: / User-agent: PageBoy Allow: / User-agent: ParaSite Allow: / User-agent: Patric Allow: / User-agent: pegasus Allow: / User-agent: The Peregrinator Allow: / User-agent: PerlCrawler 1.0 Allow: / User-agent: Phantom Allow: / User-agent: PhpDig Allow: / User-agent: PiltdownMan Allow: / User-agent: Pimptrain.com's robot Allow: / User-agent: Pioneer Allow: / User-agent: html_analyzer Allow: / User-agent: Portal Juice Spider Allow: / User-agent: PGP Key Agent Allow: / User-agent: PlumtreeWebAccessor Allow: / User-agent: Poppi Allow: / User-agent: PortalB Spider Allow: / User-agent: psbot Allow: / User-agent: GetterroboPlus Puu Allow: / User-agent: The Python Robot Allow: / User-agent: Raven Search Allow: / User-agent: RBSE Spider Allow: / User-agent: Resume Robot Allow: / User-agent: RoadHouse Crawling System Allow: / User-agent: RixBot Allow: / User-agent: Road Runner: The ImageScape Robot Allow: / User-agent: Robbie the Robot Allow: / User-agent: ComputingSite Robi/1.0 Allow: / User-agent: RoboCrawl Spider Allow: / User-agent: RoboFox Allow: / User-agent: Robozilla Allow: / User-agent: Roverbot Allow: / User-agent: RuLeS Allow: / User-agent: SafetyNet Robot Allow: / User-agent: Scooter Allow: / User-agent: Sleek Allow: / User-agent: Search.Aus-AU.COM Allow: / User-agent: SearchProcess Allow: / User-agent: Senrigan Allow: / User-agent: SG-Scout Allow: / User-agent: ShagSeeker Allow: / User-agent: Shai'Hulud Allow: / User-agent: Sift Allow: / User-agent: Simmany Robot Ver1.0 Allow: / User-agent: Site Valet Allow: / User-agent: Open Text Index Robot Allow: / User-agent: SiteTech-Rover Allow: / User-agent: Skymob.com Allow: / User-agent: SLCrawler Allow: / User-agent: Inktomi Slurp Allow: / User-agent: Smart Spider Allow: / User-agent: Snooper Allow: / User-agent: Solbot Allow: / User-agent: Spanner Allow: / User-agent: Speedy Spider Allow: / User-agent: spider_monkey Allow: / User-agent: SpiderBot Allow: / User-agent: Spiderline Crawler Allow: / User-agent: SpiderMan Allow: / User-agent: SpiderView(tm) Allow: / User-agent: Spry Wizard Robot Allow: / User-agent: Site Searcher Allow: / User-agent: Suke Allow: / User-agent: suntek search engine Allow: / User-agent: Sven Allow: / User-agent: Sygol Allow: / User-agent: TACH Black Widow Allow: / User-agent: Tarantula Allow: / User-agent: tarspider Allow: / User-agent: Tcl W3 Robot Allow: / User-agent: TechBOT Allow: / User-agent: Templeton Allow: / User-agent: TeomaTechnologies Allow: / User-agent: TITAN Allow: / User-agent: TitIn Allow: / User-agent: The TkWWW Robot Allow: / User-agent: UCSD Crawl Allow: / User-agent: UdmSearch Allow: / User-agent: UptimeBot Allow: / User-agent: URL Check Allow: / User-agent: URL Spider Pro Allow: / User-agent: Valkyrie Allow: / User-agent: Verticrawl Allow: / User-agent: Victoria Allow: / User-agent: vision-search Allow: / User-agent: void-bot Allow: / User-agent: Voyager Allow: / User-agent: VWbot Allow: / User-agent: The NWI Robot Allow: / User-agent: W3M2 Allow: / User-agent: WallPaper (alias crawlpaper) Allow: / User-agent: the World Wide Web Wanderer Allow: / User-agent: w@pSpider by wap4.com Allow: / User-agent: WebBandit Web Spider Allow: / User-agent: WebCatcher Allow: / User-agent: WebCopy Allow: / User-agent: webfetcher Allow: / User-agent: The Webfoot Robot Allow: / User-agent: Webinator Allow: / User-agent: weblayers Allow: / User-agent: WebLinker Allow: / User-agent: WebMirror Allow: / User-agent: The Web Moose Allow: / User-agent: WebQuest Allow: / User-agent: Digimarc MarcSpider Allow: / User-agent: WebReaper Allow: / User-agent: webs Allow: / User-agent: Websnarf Allow: / User-agent: WebSpider Allow: / User-agent: WebVac Allow: / User-agent: webwalk Allow: / User-agent: WebWalker Allow: / User-agent: WebWatch Allow: / User-agent: Wget Allow: / User-agent: whatUseek Winona Allow: / User-agent: WhoWhere Robot Allow: / User-agent: Wired Digital Allow: / User-agent: Weblog Monitor Allow: / User-agent: w3mir Allow: / User-agent: WebStolperer Allow: / User-agent: The Web Wombat Allow: / User-agent: The World Wide Web Worm Allow: / User-agent: WWWC Ver 0.2.5 Allow: / User-agent: WebZinger Allow: / User-agent: XGET Allow: / User-agent: Ask Jeeves/Teoma Allow: / User-agent: Architext spider Allow: / User-agent: FAST-WebCrawler Allow: / User-agent: Slurp Allow: / User-agent: ia_archiver Allow: / User-agent: Scooter Allow: / User-agent: crawler@fast Allow: / User-agent: Crawler Allow: / User-agent: InfoSeek sidewinder Allow: / User-agent: Lycos_Spider_(T-Rex) Allow: / User-agent: ia_archiver Allow: / User-agent: ABACHOBot Allow: / User-agent: Accoona-AI-Agent Allow: / User-agent: AnyApexBot Allow: / User-agent: Arachmo Allow: / User-agent: B-l-i-t-z-B-O-T Allow: / User-agent: Baiduspider Allow: / User-agent: BecomeBot Allow: / User-agent: Bimbot Allow: / User-agent: BlitzBOT Allow: / User-agent: boitho.com-dc Allow: / User-agent: boitho.com-robot Allow: / User-agent: btbot Allow: / User-agent: Cerberian Drtrs Allow: / User-agent: Charlotte Allow: / User-agent: ConveraCrawler Allow: / User-agent: cosmos Allow: / User-agent: DataparkSearch Allow: / User-agent: DiamondBot Allow: / User-agent: Discobot Allow: / User-agent: Dotbot Allow: / User-agent: EmeraldShield.com WebBot Allow: / User-agent: envolk[ITS]spider Allow: / User-agent: EsperanzaBot Allow: / User-agent: Exabot Allow: / User-agent: FAST Enterprise Crawler Allow: / User-agent: FAST-WebCrawler Allow: / User-agent: FDSE robot Allow: / User-agent: FindLinks Allow: / User-agent: FurlBot Allow: / User-agent: FyberSpider Allow: / User-agent: g2crawler Allow: / User-agent: Gaisbot Allow: / User-agent: GalaxyBot Allow: / User-agent: genieBot Allow: / User-agent: Gigabot Allow: / User-agent: Girafabot Allow: / User-agent: hl_ftien_spider Allow: / User-agent: htdig Allow: / User-agent: ichiro Allow: / User-agent: IRLbot Allow: / User-agent: IssueCrawler Allow: / User-agent: Jyxobot Allow: / User-agent: LapozzBot Allow: / User-agent: Larbin Allow: / User-agent: LinkWalker Allow: / User-agent: lmspider Allow: / User-agent: lwp-trivial Allow: / User-agent: mabontland Allow: / User-agent: magpie-crawler Allow: / User-agent: MJ12bot Allow: / User-agent: Mnogosearch Allow: / User-agent: mogimogi Allow: / User-agent: MojeekBot Allow: / User-agent: Morning Paper Allow: / User-agent: msnbot Allow: / User-agent: MSRBot Allow: / User-agent: MVAClient Allow: / User-agent: NetResearchServer Allow: / User-agent: NG-Search Allow: / User-agent: nicebot Allow: / User-agent: noxtrumbot Allow: / User-agent: Nusearch Spider Allow: / User-agent: NutchCVS Allow: / User-agent: obot Allow: / User-agent: oegp Allow: / User-agent: OmniExplorer_Bot Allow: / User-agent: OOZBOT Allow: / User-agent: Orbiter Allow: / User-agent: PageBitesHyperBot Allow: / User-agent: polybot Allow: / User-agent: Pompos Allow: / User-agent: Psbot Allow: / User-agent: PycURL Allow: / User-agent: RAMPyBot Allow: / User-agent: RufusBot Allow: / User-agent: SandCrawler Allow: / User-agent: SBIder Allow: / User-agent: Scrubby Allow: / User-agent: SearchSight Allow: / User-agent: Seekbot Allow: / User-agent: semanticdiscovery Allow: / User-agent: Sensis Web Crawler Allow: / User-agent: Shim-Crawler Allow: / User-agent: ShopWiki Allow: / User-agent: Shoula robot Allow: / User-agent: silk Allow: / User-agent: Snappy Allow: / User-agent: sogou spider Allow: / User-agent: Speedy Spider Allow: / User-agent: Sqworm Allow: / User-agent: StackRambler Allow: / User-agent: SurveyBot Allow: / User-agent: SynooBot Allow: / User-agent: Teoma Allow: / User-agent: TerrawizBot Allow: / User-agent: TheSuBot Allow: / User-agent: Thumbnail.CZ robot Allow: / User-agent: TinEye Allow: / User-agent: TurnitinBot Allow: / User-agent: updated Allow: / User-agent: Vagabondo Allow: / User-agent: VoilaBot Allow: / User-agent: Vortex Allow: / User-agent: voyager Allow: / User-agent: VYU2 Allow: / User-agent: webcollage Allow: / User-agent: Websquash.com Allow: / User-agent: wf84 Allow: / User-agent: WoFindeIch Robot Allow: / User-agent: Xaldon_WebSpider Allow: / User-agent: yacy Allow: / User-agent: Yahoo! Slurp China Allow: / User-agent: YahooSeeker Allow: / User-agent: YahooSeeker-Testing Allow: / User-agent: YandexBot Allow: / User-agent: yoogliFetchAgent Allow: / User-agent: Zao Allow: / User-agent: Zealbot Allow: / User-agent: zspider Allow: / User-agent: ZyBorg Allow: / User-agent: AbachoBOT Allow: / User-agent: abcdatos_botlink Allow: / User-agent: AESOP_com_SpiderMan Allow: / User-agent: ah-ha.com crawler Allow: / User-agent: ia_archiver Allow: / User-agent: Scooter Allow: / User-agent: Mercator Allow: / User-agent: Scooter2_Mercator_3-1.0 Allow: / User-agent: roach.smo.av.com-1.0 Allow: / User-agent: Tv_Merc_resh_26_1_D-1.0 Allow: / User-agent: AltaVista-Intranet Allow: / User-agent: FAST-WebCrawler Allow: / User-agent: Wget Allow: / User-agent: Acoon Robot Allow: / User-agent: antibot Allow: / User-agent: Atomz Allow: / User-agent: AxmoRobot Allow: / User-agent: Buscaplus Robi Allow: / User-agent: CanSeek/ Allow: / User-agent: ChristCRAWLER Allow: / User-agent: Clushbot Allow: / User-agent: Crawler Allow: / User-agent: DaAdLe.com ROBOT/ Allow: / User-agent: RaBot Allow: / User-agent: Agent-admin/ Allow: / User-agent: RaBot Allow: / User-agent: Agent-admin/ Allow: / User-agent: DeepIndex Allow: / User-agent: DittoSpyder Allow: / User-agent: Jack Allow: / User-agent: EARTHCOM.info Allow: / User-agent: Speedy Spider Allow: / User-agent: ArchitextSpider Allow: / User-agent: EuripBot Allow: / User-agent: Arachnoidea Allow: / User-agent: EZResult Allow: / User-agent: Fast PartnerSite Crawler Allow: / User-agent: FAST Data Search Crawler Allow: / User-agent: FAST Data Search Document Retriever Allow: / User-agent: KIT-Fireball Allow: / User-agent: france.misesajour.com Allow: / User-agent: FyberSearch Allow: / User-agent: GalaxyBot Allow: / User-agent: geckobot Allow: / User-agent: GenCrawler Allow: / User-agent: GeonaBot Allow: / User-agent: getRAX Allow: / User-agent: moget/2.0 Allow: / User-agent: Aranha Allow: / User-agent: Slurp.so/1.0 Allow: / User-agent: Slurp/2.0j Allow: / User-agent: Slurp/2.0-KiteHourly Allow: / User-agent: Slurp/2.0-OwlWeekly Allow: / User-agent: Slurp/3.0-AU Allow: / User-agent: Toutatis 2.5-2 Allow: / User-agent: Hubater Allow: / User-agent: IlTrovatore-Setaccio Allow: / User-agent: IncyWincy Allow: / User-agent: UltraSeek Allow: / User-agent: InfoSeek Sidewinder Allow: / User-agent: Mole2/1.0 Allow: / User-agent: MP3Bot Allow: / User-agent: C-PBWF-ip3000.com-crawler Allow: / User-agent: ip3000.com-crawler Allow: / User-agent: Knowledge.com/ Allow: / User-agent: kuloko-bot/0.2 Allow: / User-agent: LNSpiderguy Allow: / User-agent: Linknzbot Allow: / User-agent: lookbot Allow: / User-agent: MantraAgent Allow: / User-agent: NetResearchServer Allow: / User-agent: Lycos_Spider_(T-Rex) Allow: / User-agent: JoocerBot Allow: / User-agent: HenryTheMiragoRobot Allow: / User-agent: MojeekBot Allow: / User-agent: mozDex/ Allow: / User-agent: MSNBOT/0.1 Allow: / User-agent: Navadoo Crawler Allow: / User-agent: Gulliver Allow: / User-agent: ObjectsSearch/0.01 Allow: / User-agent: OnetSzukaj/ Allow: / User-agent: PicoSearch/ Allow: / User-agent: PJspider Allow: / User-agent: DIIbot Allow: / User-agent: nttdirectory_robot Allow: / User-agent: griffon Allow: / User-agent: Spider/maxbot.com Allow: / User-agent: gazz/1.0 Allow: / User-agent: NationalDirectory-SuperSpider Allow: / User-agent: dloader(NaverRobot)/ Allow: / User-agent: dumrobo(NaverRobot)/ Allow: / User-agent: noxtrumbot/ Allow: / User-agent: Openfind piranha,Shark Allow: / User-agent: Openbot/ Allow: / User-agent: psbot Allow: / User-agent: CrawlerBoy Pinpoint.com Allow: / User-agent: user.ip3000.com Allow: / User-agent: QweeryBot Allow: / User-agent: AlkalineBOT Allow: / User-agent: StackRambler/ Allow: / User-agent: SeznamBot Allow: / User-agent: Search-10 Allow: / User-agent: Fluffy the spider Allow: / User-agent: Scrubby/ Allow: / User-agent: asterias Allow: / User-agent: speedfind ramBot xtreme Allow: / User-agent: Kototoi/0.1 Allow: / User-agent: SearchByUsa Allow: / User-agent: Searchspider/ Allow: / User-agent: SightQuestBot/ Allow: / User-agent: Spider_Monkey/ Allow: / User-agent: Surfnomore Spider v1.1 Allow: / User-agent: teoma_agent1 Allow: / User-agent: Teradex_Mapper Allow: / User-agent: ESISmartSpider Allow: / User-agent: Spider TraficDublu Allow: / User-agent: Tutorial Crawler Allow: / User-agent: updated/0.1beta Allow: / User-agent: UK Searcher Spider Allow: / User-agent: Vivante Link Checker Allow: / User-agent: appie Allow: / User-agent: Nazilla Allow: / User-agent: marvin/infoseek Allow: / User-agent: MuscatFerret Allow: / User-agent: WhizBang! Lab Allow: / User-agent: ZyBorg Allow: / User-agent: WIRE WebRefiner: Allow: / User-agent: WSCbot Allow: / User-agent: Yandex Allow: / User-agent: Yellopet-Spider Allow: / User-agent: Findexa Crawler Allow: / User-agent: YBSbot search engine indexer Allow: / User-agent: libwww-perl Allow: / User-agent: Iron33 Allow: /