# Avant One Marketing Site (avantone.io) # This robots.txt applies to the marketing domain only. # The app domain (app.avantone.io) has separate noindex handling via meta tags. # ============================================================================= # MARKETING SITE (avantone.io) - FULLY INDEXABLE # ============================================================================= # Google User-agent: Googlebot Allow: / Crawl-delay: 1 User-agent: Googlebot-Image Allow: / # Bing User-agent: Bingbot Allow: / Crawl-delay: 1 # Social platforms User-agent: Twitterbot Allow: / User-agent: facebookexternalhit Allow: / User-agent: LinkedInBot Allow: / User-agent: WhatsApp Allow: / # Other search engines User-agent: Slurp Allow: / User-agent: DuckDuckBot Allow: / User-agent: Applebot Allow: / # Job aggregator bots (for public careers pages) User-agent: IndeedBot Allow: /careers Allow: /jobs/ User-agent: CareerjetBot Allow: /careers Allow: /jobs/ User-agent: Jooble Allow: /careers Allow: /jobs/ User-agent: TalentBot Allow: /careers Allow: /jobs/ User-agent: JobbotsSpider Allow: /careers Allow: /jobs/ # AI/LLM crawlers - allow indexing of marketing content User-agent: GPTBot Allow: / User-agent: ChatGPT-User Allow: / User-agent: OAI-SearchBot Allow: / User-agent: Anthropic-AI Allow: / User-agent: Claude-Web Allow: / User-agent: PerplexityBot Allow: / User-agent: Cohere-ai Allow: / # Default: allow all other bots User-agent: * Allow: / # ============================================================================= # DISALLOWED PATHS (authentication, admin, API) # ============================================================================= # These paths should not be indexed even on the marketing site Disallow: /login Disallow: /register Disallow: /reset-password Disallow: /accept-invitation Disallow: /dashboard Disallow: /settings Disallow: /api/ Disallow: /_next/ Disallow: /admin/ # ============================================================================= # SITEMAPS # ============================================================================= # Submit this one to Search Console — it lists every sitemap below plus one # per org's job-site subdomain. Sitemap: https://www.avantone.io/sitemap-index.xml # Marketing content (content/**): industries, platform, solutions, compare, # blog, insights. Generated at build time by generate-sitemap-pages.mjs. Sitemap: https://www.avantone.io/sitemap-pages.xml # Job listings for the base marketing-site job board. Sitemap: https://www.avantone.io/sitemap.xml # ============================================================================= # HOST DIRECTIVE # ============================================================================= # Preferred domain for indexing Host: https://www.avantone.io