When Should You Return 404 Instead of 403 HTTP Status Code?

Gandalf at the forbidden pass

When talking about web application security, one common denominator that repeatedly comes up is the act of disclosing sensitive application data.  Whether deep diving into OWASP’s Top 10 security flaws or as a direct topic, it is a security area that surfaces over and over again.  Sometimes the divulged information only fuels the profile the malicious user is assembling against your application or the information directly exposes a serious security weakness.

In my last post on Security Misconfiguration, part of the discussion was on properly handling error messages to ensure we don’t expose sensitive data to our users.  But it was obvious that in certain circumstances we can inadvertently disclose information that a malicious user could use to their advantage just by returning the real HTTP status code.  One of those situations is when the resource is forbidden (403).

An authorized user has requested a forbidden resource in which they receive a HTTP 403 forbidden response to the request is a common scenario.  However, by returning the applicable and valid 403, we have also made it clear that the resource does exist.  The disclosure of the resource might only provide a piece of the profile puzzle a malicious user was assembling or it might actually directly provide the user opportunities.  For example, a valid case could be that a 3rd party web resource (.axd) had a known security flaw and could be taken advantage of through other side channels when knowing the resource exists.

Though, security through obscurity is never in of itself reliable security, it can be and should be used as part of any security in-depth approach.  We can leverage this approach when we determine that it would be better to not disclose the existence of a resource, but return 404 instead of 403 HTTP status code.  Unfortunately, if you have ever attempting to do so, you will have found it less than possible and far from easy.  Maybe you’ve even given up on the attempt.  However, there is a way.

 

The Stage

Glimpse is a diagnostic tool that many of you are probably very familiar with.  One the great qualities of it are the at-your-finger-tips heads-up display that it uses to provide diagnostic information on requests.  A second favorable feature is the ability provides access controls to Glimpse’s control panel through security policies programmatically.

Unfortunately, when you turn off Glimpse either due to lack of authorization or simple because it isn’t available, navigating to its web resource /Glimpse.axd you are presented with the following error:

standard-glimpse-403

 

Unfortunately, whether we don’t have authorization or the resource has been made unavailable (turned off) we have been notified that the resource does exists, the server completely understands our request, but for us the resource is forbidden with a returned 403 HTTP status code.

403 Forbidden: The server understood the request, but is refusing to fulfill it. Authorization will not help and the request SHOULD NOT be repeated. If the request method was not HEAD and the server wishes to make public why the request has not been fulfilled, it SHOULD describe the reason for the refusal in the entity. If the server does not wish to make this information available to the client, the status code 404 (Not Found) can be used instead.

 

Now, in our case we decide that we don’t want to reveal that indeed the resource is available to anyone who doesn’t have access. It doesn’t have to be the Glimpse.axd resource as in our example, it simply might be some resource that we want to be available to only those who are authorized.  But for those less fortunate to have access, we want to make it less conspicuous.   It’s also apparent that this was a foreseen need when you read that last sentence of the 403 definition: “if the server does not wish to make this information available to the client, the status code 404 (Not Found) can be used instead”.  For reference, “this information” refers to the information on why the request was not fulfilled.

In order to have a working solution we need to meet the following criteria:

  1. Return a custom error, specifically one that conveys the requested resource does not exists
  2. Clearly return a complementing 404 HTTP status code to back up the custom error instead of the 403
  3. Retain normal 404 behavior

 

The Problem

We can attempt to have our application (ASP.NET) intercept the 403 error returned by Glimpse by specifying a custom error in web.config under <system.web>:

<system.web>
    <customErrors mode="On" defaultRedirect="~/Errors">
        <error statusCode="403" redirect="Errors/NotFound" />
    </customErrors>
</system.web>

 

But you will quickly find that your error is not being caught by ASP.NET that indeed it’s bubbling up to your host provider (e.g. IIS) as we are still seeing the same error and HTTP response status.

standard-glimpse-403

 

So we specify an IIS custom error in <system.webServer> to do a redirect

<system.webServer>
    <httpErrors errorMode="Custom" existingResponse="Replace">
        <remove statusCode="403"/>
        <error statusCode="403" path="NotFound.html" responseMode="Redirect"/>
    </httpErrors>
</system.webServer>

 

 

And we are now seeing our static HTML file served:

glimpse-iis-302-redirect-example

 

So we are now serving up our custom error and we have eliminated the 403 HTTP status code and instead returning a 302 HTTP status code.  However, this really doesn’t change anything in regards to believing the Glimpse.axd resource doesn’t exist.  We simply have said that it exists with a different HTTP status code.

So what can we do about this?

If you’re liking this, follow me on Twitter

 

The Solution

We have been able to meet our first goal of returning a custom error and part of #2 by not returning a 403.  So let’s see what we can do about returning a 404 HTTP status code instead of the 302.

Assuming you have a route for errors such as an Error controller with an appropriate Action /Error/NotFound that can return your custom Not Found error, we can tell IIS to execute the URL:

<system.webServer>
    <httpErrors errorMode="Custom">
        <remove statusCode="403" />
        <error statusCode="403" path="/Errors/NotFound" responseMode="ExecuteURL" />
    </httpErrors>
</system.webServer>

 

But the required second step is that we have to set the HTTP status code within our rendered view (e.g. Razor view).  If not, the original 403 will simply pass through and be the returned HTTP status code.

@{
    Response.StatusCode = 404;
}

<div>
    <h1>Not Found dude</h1>
</div>

 

Now we can see that our application consistent custom error is being returned along with a 404 status code. We are also preserving the requested resource which for a lot of people is a sticking point when we are redirecting to another route and losing the original requested context.

glimpse-404-returned-example

 

Bonus Round

Unfortunately, there might be times that this behavior conflicts with another similar missing resource handling scenario, or simply you want to isolate this behavior for a specific resource or tailor it differently (special hoop jumping?).  You can do that, by utilizing the <location> element in your web.config and provide the above scenario for a specific resource:

 

<configuration>
    <location path="glimpse.axd">
        <system.webServer>
            <httpErrors errorMode="Custom">
                <remove statusCode="403" />
                <error statusCode="403" path="/Errors/NotFound" responseMode="ExecuteURL" />
            </httpErrors>
        </system.webServer>
    </location>
</configuration>

 

 

Conclusion

Many will argue whether you should not be returning the existing HTTP error code such as the 403 as in our case. Usually, arguing that subsequently a 403 HTTP status code should not directly affect security.  In many ways I can completely understand where they are coming from…..in a world where malicious users don’t lurk around every corner.  Unfortunately, that is not the world we live in and if obfuscating the existence of a resource just makes it a bit tougher or discourages some malicious users, then these are precautions that should be considered especially when there are ways to obtain these additional layers of security.

About the author

Max McCarty

Max McCarty is a software developer with a passion for breathing life into big ideas. He is the founder and owner of LockMeDown.com and host of the popular Lock Me Down podcast.