# All robots will spider the domain User-agent: * Allow:/index.htm # All robots will spider the domain User-agent: * Allow:/sitemap.txt