User-agent: Scrapy Disallow: / User-agent: * Disallow: /promotions/* Sitemap: https://www.ge-tracker.com/sitemap.xml