# All search engine robots will spider the domain and all its files using User-agent: * # If you dont want to allow files being spidered use Disallow: filename.htm User-agent: * Disallow: /_private/ Disallow: /_vti_bin/ Disallow: /_vti_cnf/ Disallow: /_vti_log/ Disallow: /_vti_pvt/ Disallow: /_vti_txt/ Disallow: /skabeloner/