카테고리 없음
robots.txt 설정
냐옴
2023. 9. 15. 16:01
disallow all
User-agent: *
Disallow: /
allow all
- robots.txt 파일이 없거나
- robots.txt 파일 내용이 비었거나
- 아래 처럼 설정
User-agent: *
Disallow:
특정 디렉토리나 파일을 disallow
User-agent: *
Disallow: /foo/
Disallow: /bar/
Disallow: /hidden/file.html
참고
https://searchfacts.com/robots-txt-allow-disallow-all/
How to Use Robots.txt to Allow or Disallow Everything
The robots.txt file controls how search engine robots and web crawlers access your site. It is very easy to either allow or disallow all access.
searchfacts.com