# Allow access to RLG spider User-agent: RLGEADSpider Disallow: # User-agent: Googlebot # Allow: / # Dynaweb generates HTML on demand and does not contain HTML files User-agent: * Disallow: /