13

https://stackexchange.com/feeds/questions hasn't been updated for 12 hours and https://stackexchange.com/feeds/questions for two days. Both protocols return different content, but neither are up to date.

However, when logged in, both versions give the same content, which looks like the correct one. The <updated> field in <feed> is just minutes ago.

Diff of response headers for https between logged out and logged in (http has similar values):

Header Logged out Logged in ------------------------------------------------------------------------------------------- Cache-Control public, max-age=63626889542 public, no-cache="Set-Cookie", max-age=120 Content-Length 37765 36807 Expires Tue, 04 Apr 2017 07:59:02 GMT Wed, 05 Apr 2017 19:15:55 GMT Last-Modified Tue, 04 Apr 2017 07:57:02 GMT Wed, 05 Apr 2017 19:13:55 GMT Age 126879 (none) X-Served-By cache-yyz8335-YYZ cache-yyz8332-YYZ X-Cache HIT MISS X-Cache-Hits 4 0 X-Timer S1491419523.730773,VS0,VE1 S1491419635.717309,VS0,VE102

Edit: The plot thickens: I get the same thing on my phone on data and on two machines on my network (two different ISPs), but both a friend who lives in the same city and Cai in the comments below get an up-to-date version. I have no idea what's going on.

stackexchange.com resolves to 151.101.193.69 on both ISPs, but friend's gives 151.101.65.69. We both flushed the DNS cache and now we're both getting 151.101.1.69, but it's still giving me an outdated version.

Going through a VPN fixes all the issues, which is not that surprising, but turning my VPN off and doing a ctrl-f5 gives me back the outdated version. And like I said above, sending my session cookie always gives me the correct, up-to-date version:

$ curl -I -b "cookie string copied from browser" https://stackexchange.com/feeds/questions HTTP/1.1 200 OK Cache-Control: public, no-cache="Set-Cookie", max-age=120 Content-Length: 36558 Content-Type: application/atom+xml; charset=utf-8 Expires: Thu, 06 Apr 2017 18:50:22 GMT Last-Modified: Thu, 06 Apr 2017 18:48:22 GMT <----- Note X-Frame-Options: SAMEORIGIN Set-Cookie: (redacted) X-Request-Guid: c9482245-8d6b-4de5-a8de-1e25205e54c9 Accept-Ranges: bytes Date: Thu, 06 Apr 2017 18:48:22 GMT Via: 1.1 varnish Connection: keep-alive X-Served-By: cache-yyz8330-YYZ X-Cache: MISS X-Cache-Hits: 0 X-Timer: S1491504502.382361,VS0,VE51 Vary: *,Fastly-SSL X-DNS-Prefetch-Control: off $ curl -I https://stackexchange.com/feeds/questions HTTP/1.1 200 OK Cache-Control: public, max-age=63626889542 Content-Type: application/atom+xml; charset=utf-8 Expires: Tue, 04 Apr 2017 07:59:02 GMT Last-Modified: Tue, 04 Apr 2017 07:57:02 GMT <----- Note X-Frame-Options: SAMEORIGIN X-Request-Guid: 2087959f-edc8-4b74-a078-8bcb874b4337 Content-Length: 37765 Accept-Ranges: bytes Date: Thu, 06 Apr 2017 18:48:37 GMT Via: 1.1 varnish Age: 211874 Connection: keep-alive X-Served-By: cache-yyz8335-YYZ X-Cache: HIT X-Cache-Hits: 7 X-Timer: S1491504518.899367,VS0,VE0 Vary: *,Fastly-SSL X-DNS-Prefetch-Control: off Set-Cookie: (redacted)
Improve this question
11
  • I've only checked the HNQ, I don't know if the other feeds have issues. Commented Apr 5, 2017 at 19:43
  • Guess it's the cache control max-age, which is set to, like... forever Commented Apr 5, 2017 at 19:51
  • @Cai It might interfere with normal browser requests, but curl, wget or just ctrl-f5 wouldn't care. They all fail right now. Commented Apr 6, 2017 at 7:38
  • well fwiw they're all updating for me (last-modified is always a within the last few minutes at least, I'm not actually checking the content...) Commented Apr 6, 2017 at 8:24
  • @Cai Are you sure? They're not for me. Can you check in a private session in your browser to make sure you're not logged in? Commented Apr 6, 2017 at 16:56
  • 1
    Both in a private session, definitely not logged in: HTTPS and HTTP, I've checked a couple times today and it seems to always been up to date Commented Apr 6, 2017 at 17:14
  • Actually, just noticed the max-age is looking a bit more reasonable on HTTPS now; I was getting the same as you on that earlier (although it was still up to date) Commented Apr 6, 2017 at 17:16
  • Yeh no idea, very strange... hopefully a dev will have a look and see what's going on Commented Apr 6, 2017 at 17:35
  • Thanks for the heads up, we're taking a look at the CDN here. Commented Apr 7, 2017 at 16:50
  • @isanae Just to help us save some dig time here - any idea when this started? You seem to be an avid user, so is this very recent? We haven't made any known changes here...so digging into the unknown. Commented Apr 7, 2017 at 17:14
  • @NickCraver I've opened a chat room. Commented Apr 7, 2017 at 17:16

1 Answer 1

Reset to default
21

First, for all website admins using sites with output caching:

If you use [OutputCache], DO NOT UPGRADE TO .NET 4.6.2

Details:

Unfortunately, this was far larger than your feed and we're still assessing the damage. There is a critical bug in the 4.6.2 release of the .NET Framework which causes cache poisoning in a severe fashion. Though the cause is a simple errant setter, the impact is tremendous. Here's the fix in .NET 4.7 (not available for our Windows Server edition yet): .NET 4.6.2 vs 4.7 (You're looking at HttpCachePolicy.cs, here's the link for .NET 4.7)

The manifestation of this bug is an errant setting _utcTimestampRequest to 0, and amounts to running the following to calculate the max-age:

0 - (DateTime.MinValue - DateTime.UtcNow).TotalSeconds

Which is 63,627,196,173 seconds, or about 2017 years (since it's the diff to DateTime.MinValue or 0001-01-01 00:00:00Z in .NET).

This means browsers and proxies are instructed to cache content not for the 15 to 180 seconds we instructed, but 2 millennia instead. This is a huge, huge bug that needs a hotfix for all customers ASAP. I am stressing that to Microsoft as strongly as we can.

While we are rather agile and will deploy a private .NET 4.7 build if we can arrange one, most people aren't so lucky and often have to wait many months before they can update their releases, whereas a minor 4.6.x release can happen quickly. Your only options (currently) are to roll forward, roll back, or ditch output caching. We've currently opted for #3.

We have disabled all output cache directives at Fastly and stripped the rest in code. The only exception to this is user flair (a bit heavier on bytes), which we're manually overriding in Fastly to the 180 seconds it should be.

I will continue to work with Microsoft on a resolution into the evening and update this as I am able.

Improve this answer
1
  • Do you happen to remember what happened after that? :) Commented Feb 4, 2024 at 13:52

You must log in to answer this question.