Two hard to count the inbox notifications from chat

This was a fun one. Fixed now.

As I mentioned in a comment, this was related to the migration of our auth service to GCP. It's the first thing we've moved that actually does non-trivial backend work (StackSnippets has been in GCP for a while, but it doesn't talk to things like Redis or databases).

This auth service, called "StackAuth", is responsible for network login functionality, but that's not the only thing it does. For largely historical reasons, it also has some small private APIs for "global network things", i.e. things that don't necessarily involve a specific Stack Exchange site.

One of those things is a private API that Chat calls to insert items into the Stack Exchange™ MultiCollider SuperDropdown™, a.k.a. the global inbox.

This involves three things:

1. Inserting the inbox item into the database,
2. sending a message via Redis that gets forwarded to the websocket, so that your inbox lights up immediately when you have a new notification, and
3. incrementing a counter in Redis that stores your unread notification count, which is how the site still knows about that number on subsequent page loads.

Points 1 and 2 worked fine -- the notification was actually in your inbox (point 1), and the realtime notification also came as expected. But point 3 didn't work -- when you loaded the page after the message appeared in your inbox, you didn't get an indication of that, because the counter had never been incremented.

One detail you need to now about our cloud migration is that we start by only migrating application itself, while keeping shared resources like SQL databases and Redis in our datacenter initially. That means that for now, StackAuth runs happily in a Kubernetes cluster in GCP, but it still talks to SQL Server and Redis on bare metal servers in New York New Jersey.

And Redis in particular was the culprit here. Clearly, Redis kinda worked -- that's point 2 above. So what's different about point 3?

Well, the realtime notifications use Redis pub/sub channels, but the counter from point 3 is an actual value in the redis key space.

For the non-pub/sub Redis use, we utilize a Redis feature called "databases", which are basically namespaces. Redis' creator has said that the databases feature was a mistake, but that didn't stop us from using them 😬.

Each Stack Exchange site has a database for any site-specific caching needs, and there's one database (number 0) for all the network-wide stuff -- like, for example, inboxes.

Now, because Redis databases are a "please don't use this" feature, we're gradually stopping its use. Stack Overflow for Teams switched to a different approach a while ago, where we only use database 0, and instead differentiate individual sites by using a key prefix (or a "g" prefix for global stuff).

For the public network, we are finally going to make that change as well as we're migrating to GCP. And that's why the application in GCP was already configured to use the new model. But this was wrong, because it still talks to Redis in the datacenter, together with other on-prem apps.

And so, when a new chat inbox message came in, StackAuth in GCP incremented a counter with a "g" prefix, but the Q&A website (which displays your inbox) read a counter without the prefix.

This misconfiguration is fixed now -- switching everything to use the new Redis model is going to be a later step.