23

I am running nginx 0.6.32 as a proxy front-end for couchdb. I have my robots.txt in the database, reachable as http://www.example.com/prod/_design/mydesign/robots.txt. I also have my sitemap.xml which is dynamically generated, on a similar url.

I have tried the following config:

server { listen 80; server_name example.com; location / { if ($request_method = DELETE) { return 444; } if ($request_uri ~* "^/robots.txt") { rewrite ^/robots.txt http://www.example.com/prod/_design/mydesign/robots.txt permanent; } proxy-pass http://localhost:5984; proxy_redirect off; proxy_set_header Host $host; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; }

This appears to work as a redirect but is there a simpler way?

Improve this question

6 Answers 6

Reset to default
67

Or you can put it simply in its own location -

location /robots.txt { alias /Directory-containing-robots-file/robots.txt; }
Improve this answer
Sign up to request clarification or add additional context in comments.

6 Comments

This seems like the simplest solution, and works perfectly for me.
When you say directory-containing-file, I presume you mean including the filename? I tried alias /www/static;, with and without trailing slashes, and it failed. When I used alias /www/static/robots.txt;, it worked fine.
John C - No, only the directory. instead of putting alias /directory; you can also put root /opt/directory/; Hope that helps.
On my dev site i am using a wildcard in my sites-available to serve up any one of the projects.. can i use this approach to have all my projects use the same robots.txt file? if so then how? I tried to follow your directions but the robots.txt file that is within that project is being served
Thanks! If you need sitemaps (more than one file), location /sitemap { root /home/ec2-user/app_directory/public; }
|
20

I think you want to setup a location. The key part being the "=" directive.

location = /robots.txt { alias /your_full_path/robots.txt ; }

(When I didn't include the =, it still went to the default nginx root location)

Setting it like below, would cause all /robots.txt* requests to be read out of /var/foo. So /robots.txt.bing tries reading /var/foo/robots.txt.bing off of disk. The "^~" indicates that it is a regular expression match on the beginning of the request.

location ^~ /robots.txt { root /var/foo; }
Improve this answer

Comments

5

rewrite with http://... is inteded to do the redirect. You are probably looking for:

 rewrite ^/robots.txt /prod/_design/mydesign/robots.txt last;
Improve this answer

1 Comment

needs equals to work: location = /sitemap.xml { rewrite ^/sitemap\.xml$ /index.php?xxxxxxxxx permanent; }
3

This should work also, and may perform slightly better.

location /robots.txt { alias /var/www/django/mysite/static/robots.txt; }
Improve this answer

1 Comment

Why may it perform slightly better?
1

Note that the first parameter is a regular expression so you should probably escape . and match end of line.

location / { rewrite ^/robots\.txt$ /static/robots.txt last; ... }

NginxHttpRewriteModule

Improve this answer

Comments

0

Considering that robots.txt is located at /var/www/mysite/static/robots.txt, the following worked for me:

location = /robots.txt { root /var/www/mysite/static/; }
Improve this answer

Comments

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.