User-agent: * # NOTE ON METHOD # Pages we want OUT of the index carry a # and are deliberately NOT disallowed here: a disallowed page is never fetched, # so the crawler would never see the noindex and an already-indexed URL would # linger as a bare listing. Only paths with no HTML to carry a meta tag — the # API and the redirect endpoints — are blocked below. # API surface: nothing to index, and crawling it burns rate limit. Disallow: /api/ Disallow: /links/api/ # Link redirects and interstitials. Following these inflates click analytics # with bot traffic, which is the real reason to keep crawlers out. Disallow: /t/ Disallow: /links/t/ # AI answer engines: a plain factual summary of WizBI lives at /llms.txt. # It is allowed like every other public path; listed here so it is easy to find. Allow: /llms.txt Sitemap: https://wizbi.ai/sitemap.xml