36

When working with a resource-based site (such as an MVC application or REST service), we have two main options when a client tries to GET a resource that they don't have access to:

  • 403, which says that the client is unauthorized; or
  • 404, which says that the resource does not exist (or couldn't be located).

Common wisdom and common practice seems to be to respond with the truth - that is, a 403. But I'm wondering if this is actually the right thing to do.

Secure login systems never tell you the reason for a login failure. That is to say, as far as the client is concerned, there is no detectable difference between a non-existent user name and an incorrect password. The purpose is of this is to not make user IDs - or worse, e-mail addresses - discoverable.

From a privacy standpoint, it seems a lot safer to return a 404. I'm reminded of the incident wherein someone reportedly found out the winners of a reality show (Survivor, I think) by looking at which resources didn't exist on the site vs. which ones did. I'm concerned about a 403 potentially giving away sensitive information like a serial number or account number.

Are there compelling reasons not to return a 404? Could a 404 policy have negative side effects elsewhere? If not, then why isn't the practice more common?

Improve this question
3
  • 1
    Compelling reason not to return 404: If the service is down or there is a bug/error in authentication then you'd get a 404, but the customer/user/tester/developer/support person trying to diagnose the problem may have no idea what's wrong from the error message. Commented Jun 8, 2011 at 2:03
  • @Snorfus: That's a good point - I'd have put it in an answer. Although Josh did already add a good counterpoint... Commented Jun 8, 2011 at 2:18
  • Another aspect to keep in mind is this: how are 404s treated downstream? Is there a CDN or some kind of caching? If you ever want to be able to cache those, then you probably don't want your "user doesn't have permission" 404s to be cached. Commented Nov 26, 2011 at 21:48

4 Answers 4

Reset to default
17

There's a general misconception (and misuse) associated with 403 Forbidden: it's not supposed to give anything away about what the server thinks about the request. It's specifically designed to say,

I get what you're requesting, but I'm not going handle the request, no matter what you try. So stop trying.

Any UA or client should interpret that to mean that the request will never work, and respond appropriately.

This has implications for clients handling requests on behalf of users: if a user isn't logged in, or mistypes, the client handling the request should reply, "I'm sorry, but I can't do anything" after the first time it gets the 403 and stop handling future requests. Obviously, if you want a user to still be able to request access to their personal information after a failure, this is a user-hostile behavior.

403 is in contrast to 401 Authorization Required, which does give away that the server will handle the request as long as you pass the correct credentials. This is usually what people think about when they hear 403.

It's also in contrast with 404 Page Not Found which, as others pointed out, is designed not only to say "I can't find that page" but to suggest to the client that the server makes no claims of success or failure for future requests.

With 401 and 404, the server doesn't say anything to the client or UA about how they should proceed: they can keep trying in hopes of getting a different response.

So 404 is the appropriate way to handle a page you don't want to show to everyone, but don't want to give away anything about why you won't show it in certain situations.

Of course, this assumes the client making the request cares for petty RFC flippancy. A malicious enough client isn't going to care about the status code returned except in an incidental manner. One will know it's a hidden user page (or a potential hidden user page) by comparing it to other, known user pages.

That is, let's say your handler is users/*. If I know users/foo, users/bar and users/baaz work, the server returning a 401, 403, or 404 for users/quux doesn't mean I'm not going to try it, especially if I have reason to believe there is a quux user. A standard example scenario is Facebook: my profile is private, but my comments on public profiles are not. A malicious client knows I exist even if you return 404 on my profile page.

So status codes aren't for the malicious use cases, they're for the clients playing by the rules. And for those clients, a 401 or a 404 request is most appropriate.

Improve this answer
4
  • 4
    Good points about 401 vs. 403. I don't agree with the finality of your statements on 404, though. Sure, in the example of Facebook, you already know that quux exists. But there's not always going to be external evidence, especially on non-social sites where client data is really private. A nosey or hostile user might eventually be able to deduce that you are lying, but not necessarily which resources you are lying about. I think the salient point here is really, don't bother to 404 resources unless there are no other methods of discovery available. Commented Jun 8, 2011 at 3:36
  • @Aaronaught The thing is, you have to be really, really sure someone can't figure out what the server's being deceptive about, or that there's not some method of discovery you haven't hardened against. A smart enough person will figure it out, and at that point, the status code is meaningless. Commented Jun 8, 2011 at 3:48
  • 1
    I'm still not clear on how a smart enough person could figure it out if there is no sharing of data between user profiles. I understand that someone determined will eventually realize "oh, a 404 actually means it might exists, but I just can't access it" - but assuming that the server returns the same error for resources which genuinely don't exist, how would one tell the difference between a non-existent resource and a privileged one? Commented Jun 8, 2011 at 14:57
  • @Aaronaught the only thing a person would need to know is the URL to a profile should exist. You might just use profile IDs and increment them in a pseudo fashion (so a person with a profile ID of 3333 doesn't mean there's a person with a profile ID of 3332), but a determined person should be able to predict it given some sample size of valid IDs. Failing that, and even given a completely hardened system that leaks no information about how it generates URLs for profile pages, there's always social engineering. Commented Jun 8, 2011 at 17:25
10

According to RFC2616

10.4.4 403 Forbidden

The server understood the request, but is refusing to fulfill it. Authorization will not help and the request SHOULD NOT be repeated. If the request method was not HEAD and the server wishes to make public why the request has not been fulfilled, it SHOULD describe the reason for the refusal in the entity. If the server does not wish to make this information available to the client, the status code 404 (Not Found) can be used instead.

also note that when accessing a Forbidden resources, authorization will not help.

Improve this answer
3
  • I am aware of the RFC, although it bears little resemblance to reality. Almost no server ever explains the reason for a 403 beyond that first sentence ("The server understood the request, but is refusing to fulfill it.") and most MVC frameworks even have something akin to an HttpUnauthorizedResult that are intended to be used for privileged resources. I also realize (obviously) that 404 can be used instead; my question is whether or not it should be used, and what should be considered when making that decision. Commented Jun 8, 2011 at 1:59
  • @Aaronaught: the RFC does not just allow you to use 404, it recommends that 404 be thrown when you don't want to acknowledge anything. Commented Jun 8, 2011 at 2:10
  • 3
    That's fine, but when should I not acknowledge anything? Or rather, when should I? That's really the essence of the question. Commented Jun 8, 2011 at 2:16
6

Should you? Yes.

You said it yourself, betray as little as possible. If I was attacking a system and noticed the server responded with 403 codes I would focus on those, instead of moving on. Better a door proclaim it doesn't exist then to proclaim it to be barred.

The downside of using 404 requests is that externally it will appear as if the page doesn't exist, and this could have conflicts when compared to pages that are supposed to exist but are missing instead. If you aren't worried about web crawlers (authenticated systems should be entirely denied anyways) then you should absolutely go for it. Every API I've handles unauthorized access the exact same way, and so does StackOverflow. Can't see that page? I assure you it does exist, even though it claims not to.

You should acknowledge requests when the resource is known to exist and access is denied. A failed login should not result in a 404 message. You shouldn't acknowledge requests when the existence of the resource itself should be protected. Access to the realm exists in the public, but the security groups or roles within it are not.

Compelling reason not to return 404: If the service is down or there is a bug/error in authentication then you'd get a 404, but the customer/user/tester/developer/support person trying to diagnose the problem may have no idea what's wrong from the error message

Developers should have access to logs, which will indicated an attempt to access a protected resource. Customers / Users / Testers will generate feedback which will eventually hit a developer.

Security by obscurity... really?

This is not security by obscurity. You are using obscurity in addition to proper security measures.

Valid requests getting a "404" on the other hand are just adding complexity and obscurity where it's not necessary

They aren't valid requests, they are unauthorized requests. You are changing a small part (return 404 instead of 403) to gain a substantial advantage in any would-be attackers.

Improve this answer
8
  • Security by obscurity... really? If someone's trying to poke your service(s) with malicious intent, they already know it's there. Valid requests getting a "404" on the other hand are just adding complexity and obscurity where it's not necessary. Commented Jun 8, 2011 at 2:29
  • 4
    @Snorfus: There's a subtle distinction to be made here between security and privacy. Privacy is, almost by definition, "by obscurity". This is especially important in the context of a resource based system, since the existence or non-existence of a resource is potentially important information. There may be security arguments as well, although I'm less concerned with those. I would like to hear more about this added complexity; can you go into detail beyond your initial comment? (Perhaps in a more comprehensive answer? Have you been bitten by 404'ed 403s?) Commented Jun 8, 2011 at 2:41
  • 2
    @Snorfus: I'd also like to add, as an afterthought, that defense in depth is not the same as security by obscurity. The latter implies that there are no other means of protection. But adding obscurity to an already-secure system can often be a good thing - as long as it doesn't cause other problems down the road. In this case, it's a given that the system is already secured, since the 404s are being emitted by authorization code. Commented Jun 8, 2011 at 3:02
  • @Aaronaught: I'll post a more in-depth answer, but on the issue: if a resource is private then what is the reasoning behind exposing it publicly? Commented Jun 8, 2011 at 3:33
  • @Snorfus: A resource is private to a client - or maybe to a group - not to the entire world. Commented Jun 8, 2011 at 3:37
1

It really depends on the information given away by the 403 status code. If you have an API endpoint responding to GET /myresource/{id} with a 404 when the resource does not exist, then the status code returned by the endpoint when the resource exists but is not authorized to the current user should depend on the predictability of the id parameter, and the amount of information it contains by itself.

  • If id is a private field like an email address or a bank account number, it probably should always be hidden behind a 404. In the case of an email address it could prevent spamming bots from compiling a list of valid email addresses registered to your website.
  • If id is a public username, don't bother, there are other means to get access to this information (like, just by browsing the forum page of your website).
  • If id is an opaque, but predictable identifier (e.g. an auto-incrementing integer), you don't give away any information to the attacker that he could not get by other means. So in my opinion it's safe to return a 403 status code.
  • If id is an opaque, unpredictable identifier (like a random GUID), the question is more subtle. I personally think there is no problem in returning a 403 status code. This information could only be exploited if there is another flawed endpoint in your API using this ID, but the real flaw IS your other endpoint.

You may assume it's better to be safe than sorry in any case and hide the information whatever the context, but you actually can't rely on this type of behavior to provide any form of security. On the other hand, it can provide confidentiality (or privacy, like others have said).

A security mechanism should not just be a speed bump for the attacker, otherwise it's just plain useless. In this case, it might even prevent you from using other features correctly (crawlers, Web Application Firewall looking for abnormal amounts of 403 status codes to block users...).

Improve this answer
1
  • I believe this answer properly conveys the nuance - it all depends on many factors. Commented May 13, 2024 at 8:40

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.