Sitemap: http://www.antispyware.ws/sitemap.xml User-agent: * Crawl-delay: 5 # Directories Disallow: /carp/ Disallow: /files/ Disallow: /images/ Disallow: /docs/ Disallow: /videos/ Disallow: /database/ Disallow: /includes/ Disallow: /misc/ Disallow: /modules/ Disallow: /sites/ Disallow: /themes/ Disallow: /scripts/ Disallow: /updates/ Disallow: /profiles/ Disallow: /user/ # Files Disallow: /CHANGELOG.txt Disallow: /cron.php Disallow: /INSTALL.mysql.txt Disallow: /INSTALL.pgsql.txt Disallow: /install.php Disallow: /INSTALL.txt Disallow: /LICENSE.txt Disallow: /MAINTAINERS.txt Disallow: /update.php Disallow: /UPGRADE.txt Disallow: /xmlrpc.php # Paths (clean URLs) Disallow: /admin/ Disallow: /comment/reply/ Disallow: /contact/ Disallow: /logout/ Disallow: /node/add/ Disallow: /search/ Disallow: /user/register/ Disallow: /user/password/ Disallow: /user/login/ # Paths (no clean URLs) Disallow: /?q=admin/ Disallow: /?q=comment/reply/ Disallow: /?q=contact/ Disallow: /?q=logout/ Disallow: /?q=node/add/ Disallow: /?q=search/ Disallow: /?q=user/password/ Disallow: /?q=user/register/ Disallow: /?q=user/login/ User-agent: Wget Disallow: / User-agent: archive.org_bot Disallow: / User-agent: Xenu's Link Sleuth 1.1c Disallow: / User-agent: Xenu's Disallow: / User-agent: Gaisbot Disallow: / User-agent: dotbot Disallow: / User-agent: OmniExplorer_Bot Disallow: / User-Agent: Charlotte Disallow: / User-agent: CazoodleBot Disallow: / User-agent: Jyxobot Disallow: / User-agent: Sogou Pic Spider Disallow: / User-agent: FreeFind Disallow: / User-agent: BecomeBot Disallow: / User-agent: Nutch Disallow: / User-agent: Jetbot Disallow: / User-agent: WebVac Disallow: / User-agent: Stanford Disallow: / User-agent: naver Disallow: / User-agent: dumbot Disallow: / User-agent: Hatena Antenna Disallow: / User-agent: grub Disallow: / User-agent: WebZip Disallow: / User-agent: larbin Disallow: / User-agent: b2w Disallow: / User-agent: psbot Disallow: / User-agent: Python Disallow: / User-agent: NetMechanic Disallow: / User-agent: URL_Spider_Pro Disallow: / User-agent: CherryPicker Disallow: / User-agent: EmailCollector Disallow: / User-agent: EmailSiphon Disallow: / User-agent: WebBandit Disallow: / User-agent: EmailWolf Disallow: / User-agent: ExtractorPro Disallow: / User-agent: CopyRightCheck Disallow: / User-agent: Crescent Disallow: / User-agent: SiteSnagger Disallow: / User-agent: ProWebWalker Disallow: / User-agent: CheeseBot Disallow: / User-agent: LNSpiderguy Disallow: / User-agent: Alexibot Disallow: / User-agent: Teleport Disallow: / User-agent: TeleportPro Disallow: / User-agent: Stanford Comp Sci Disallow: / User-agent: MIIxpc Disallow: / User-agent: Telesoft Disallow: / User-agent: Website Quester Disallow: / User-agent: moget Disallow: / User-agent: WebZip Disallow: / User-agent: WebStripper Disallow: / User-agent: WebSauger Disallow: / User-agent: WebCopier Disallow: / User-agent: NetAnts Disallow: / User-agent: Mister PiX Disallow: / User-agent: WebAuto Disallow: / User-agent: TheNomad Disallow: / User-agent: RMA Disallow: / User-agent: libWeb Disallow: / User-agent: asterias Disallow: / User-agent: httplib Disallow: / User-agent: turingos Disallow: / User-agent: spanner Disallow: / User-agent: InfoNaviRobot Disallow: / User-agent: Harvest Disallow: / User-agent: Bullseye Disallow: / User-agent: CherryPickerSE Disallow: / User-agent: CherryPickerElite Disallow: / User-agent: WebBandit Disallow: / User-agent: NICErsPRO Disallow: / User-agent: DittoSpyder Disallow: / User-agent: Foobot Disallow: / User-agent: SpankBot Disallow: / User-agent: BotALot Disallow: / User-agent: BunnySlippers Disallow: / User-agent: URLy Warning Disallow: / User-agent: LinkWalker Disallow: / User-agent: cosmos Disallow: / User-agent: moget Disallow: / User-agent: hloader Disallow: / User-agent: humanlinks Disallow: / User-agent: LinkextractorPro Disallow: / User-agent: Offline Explorer Disallow: / User-agent: Mata Hari Disallow: / User-agent: LexiBot Disallow: / User-agent: Web Image Collector Disallow: / User-agent: The Intraformant Disallow: / User-agent: True_Robot Disallow: / User-agent: BlowFish Disallow: / User-agent: JennyBot Disallow: / User-agent: MIIxpc Disallow: / User-agent: BuiltBotTough Disallow: / User-agent: ProPowerBot Disallow: / User-agent: BackDoorBot Disallow: / User-agent: toCrawl Disallow: / User-agent: WebEnhancer Disallow: / User-agent: suzuran Disallow: / User-agent: VCI Disallow: / User-agent: Szukacz Disallow: / User-agent: QueryN Metasearch Disallow: / User-agent: Openfind data gathere Disallow: / User-agent: Openfind Disallow: / User-agent: Xenu Disallow: / User-agent: RepoMonkey Disallow: / User-agent: Openbot Disallow: / User-agent: Zeus Link Scout Disallow: / User-agent: Zeus Disallow: / User-agent: Webster Pro Disallow: / User-agent: EroCrawler Disallow: / User-agent: LinkScan Disallow: / User-agent: Keyword Density Disallow: / User-agent: Kenjin Spider Disallow: / User-agent: Iron33 Disallow: / User-agent: Bookmark search tool Disallow: /