agent Tag

Lernen Sie uns kennenLernen Sie uns kennen

Die robots.txt von reddit gefällt mir. # 80legs User-agent: 008 Disallow: / User-Agent: bender Disallow: /my_shiny_metal_ass User-Agent: Gort Disallow: /earth User-Agent: * Disallow: /*.json Disallow: /*.json-compact Disallow: /*.json-html Disallow: /*.mobile Disallow: /*.compact Disallow: /*.xml Disallow: /*.rss Disallow: /*.i Disallow: /*.embed Disallow: /*/comments/*?*sort= Disallow: /r/*/comments/*/*/c* Disallow: /comments/*/*/c* Disallow: /r/*/submit Disallow: /message/compose* Disallow: /api Disallow: /post Disallow: /submit Disallow: /goto Disallow: /*after= Disallow: /*before= Disallow: /domain/*t= Disallow: /login Disallow: /reddits/search Disallow: /search Disallow: /r/*/search Allow: /  ...