Technical SEO Developer Hub

Generate a clean robots.txt file for crawlers, staging blocks, sitemap discovery, and simple SEO crawl control.

Check pasted XML sitemaps for basic structure, URL counts, loc values, lastmod format and common SEO mistakes.

Analyze pasted HTTP response headers for cache, redirects, security headers and content-type issues without making server-side requests.

Generate Nginx redirect snippets for moved pages, HTTPS upgrades, and canonical host cleanup.

Create Apache .htaccess redirect rules for URL migrations, canonical host changes, and HTTP to HTTPS cleanup.

A small Disallow rule can block valuable pages or rendering assets. Review robots.txt as a public crawl-control file, not a security layer.

Unexpected token errors mean the parser found a character that is not valid at that position. Often the input is not JSON at all.

An expired certificate may be on the origin, CDN edge, www host or a forgotten subdomain. Check the exact host users visit.

A minimal robots.txt example for public websites. It allows crawling and points crawlers to the sitemap.

A narrow path redirect example for Nginx. Test syntax and Location headers before using permanent redirects widely.

A basic Apache redirect snippet for one moved URL. Back up .htaccess before editing production rules.

Use robots.txt for crawl guidance, not security. The safest workflow is to keep public pages and assets crawlable, block low-value paths, and verify the file from the domain root.

A useful sitemap contains canonical, crawlable URLs that return 200 responses. It should not be a dump of every route your application can produce.

Redirect changes are small but risky. A safe workflow generates a narrow rule, validates Nginx syntax, tests Location headers and keeps rollback simple.