I will add a couple of points here:

FWIW, I think this introduces both a discrepancy with the logic we adopt at the edge, and a matter of inequality: bots who control a large enough IP space would have a much larger limit than the bot of a community member, which is typically coming from a single source IP.

If you want to expose the HTTP API for users, that would happen via the component the linked artifact service would expose. I don't understand what the issue is here.

The way I see it this was a case of "failure in depth".

Just my 2 cents about various objections I've seen raised in this task:

• If anything, we should've started moving to gRPC for internal service to service communications a long time ago; we haven't done it mostly because we didn't have immediate needs.
• Our mesh is not built "around HTTP" more than it's built around gRPC. It can work pefectly well with grpc. In fact, quite a few internal functions of our service mesh are built using grpc (for the same reason it would make sense using grpc here)
• Given gRPC uses HTTP/2 for transport, and it works perfectly fine with our mesh, our ingress, and our routing logic.
• I'm not sure I understand the comment about needing an http replica, but frankly, what stops you from running your "service" (which is just a lambda) with a sidecar of the linked artifact cache to provide the http interface?

In T414338#11652561, @matmarex wrote:

@Joe @CDanis I heard you're the people to talk to about the desired data and format of these query parameters.

Currently, the proposed patch https://gerrit.wikimedia.org/r/c/mediawiki/core/+/1239464 includes the following data:

• Site which is requesting the image, e.g. 'www.mediawiki.org'
• Generator (the software component involved), e.g. 'parser' or 'imageinfo'. Entry point is used as fallback if not specified, e.g. 'index', 'api', 'rest'
• Format of the requested image, 'original', 'thumbnail' or 'thumbnail_unscaled'

The format is UTM parameters (respectively utm_source, utm_campaign and utm_content, in this order), on the assumption that they'll be stripped by search engines etc.

Example: https://upload.wikimedia.org/wikipedia/commons/a/a9/Example.jpg?utm_source=mediawiki.localhost&utm_campaign=parser&utm_content=thumbnail

Your thoughts on that would be appreciated. I also have two questions:

• Do we need to sign these parameters so that they can't be spoofed, or do we start by assuming everyone will play nice? If yes, what format would be convenient? Can we just stick the data in a JWT instead of having separate parameters?
• If we're considering switching to a JWT, would it be more convenient to start with a single JSON parameter instead of separate parameters? (I mean something like https://upload.wikimedia.org/wikipedia/commons/a/a9/Example.jpg?utm_source={"site":"mediawiki.localhost","generator":"parser","format":"thumbnail"})

For now, let's remove the individual headers at the edge; we will have time to come up with a prefix naming (if any) that we want to use going forward, and to convert the headers to those values.

I don't want us to make anything under x-wmf be stripped automatically. It limits ourselves in the future.

Added an exception for translatewiki so they can keep syncing from gerrit at the previous rate, given that never created an issue for us.

Ah seen the script (thanks @Nikerabbit); first thing to do is make the user-agent compliant with the wikimedia User-Agent policy, so it should contain an email address or an url.

Do you happen to know what is the user-agent used by the script? Or point me to the source code for it.

In T364245#11635060, @tstarling wrote:

It's not using HTTPS is it?

In T402959#11542490, @Joe wrote:

In T402959#11478074, @BTracy-WMF wrote:

Hi @CDanis , we recently finalized a decision brief on next steps for this issue. We aligned upon revoking the exception and working with WDQS users to ensure their federated queries are compliant with the UA policy. To ensure we have enough lead time to communicate the upcoming change, we are requesting that the policy exception remain in place until February 18th, 2026.

Please let me know if this poses any issues on the SRE side.

Sorry, I missed this message as I was on PTO. The original agreed date was start of the year; however - February 18 is ok from our point of view, as long as there aren't further delays.

In T414805#11620283, @Ladsgroup wrote:

FWIW: Out of 9K‌ results in https://global-search.toolforge.org/?q=%5C%2Fthumb%5C%2F&regex=1&namespaces=8&title= the vast majority of them are CN banners (search for MediaWiki:Centralnotice-template) and from the remaining ones, a significant portion of them use standard sizes already.

In T414805#11612457, @Krinkle wrote:

In T414805#11612140, @gerritbot wrote:

[mediawiki/extensions/WikiEditor@master] Set background-size for toolbar buttons

https://gerrit.wikimedia.org/r/1233759

@MatthewVernon Based on discussions with SRE for T414338: FY25-26 WE5.4.12: Identify the provenance of image requests, I believe tiny icons are not our focus. Is that right? The long tail in CSS for gadgets, tools, apps, user scripts, extensions, etc to update and chase down to avoid broken UI makes this a daunting effort that I'm not sure is worth taking on righ tnow.

Could we amend the initial restriction proposed here to say that widths under 60px remain permitted? We can continue to treat these as non-standard and apply low limits to their cache misses, but we would continue to treat them as syntactically valid URLs that can be served from the cache, and at a low rate for regeneration.

Hi @Salujapushpit - I'm the project maintainer, I have seen your MR, I will review it.

I had resolved this bug quite some time ago.

Given how large the current set of actions is, this would result in a super clogged/unusable UI.

I think we have a better chance of actually integrating druid in some form within HP, rather than allowing to do this.

This has been indeed implemented already by @CDanis

Hi @Scott_French, I assume this task is resolved?

Not sure I get what you're referring to. The buttons are not javascript actions but simple submit buttons for HTML forms. They shouldn't be clickable more than once. Without reproductions steps I don't know what to do with this bug.

Thank you for tagging this task with good first task for Wikimedia newcomers!

I should add - it's also possible that actually huggle doesn't send auth cookies with every request; in that case, you get rate-limited to 100 requests over 10 seconds, unless you're using the tool from toolsforge.

Ah sorry, you're saying you're using bot-password above. I missed that. Uhm, this means that probably the hotfix I deployed to try to hotpatch the issue while T415007 is resolved isn't working. I'll investigate further tomorrow to see if my hotfix can indeed work.

@Framawiki the error message you get seems to indicate that your tool is not authenticated, but the UA is correctly detected as a bot:

In T402959#11478074, @BTracy-WMF wrote:

Hi @CDanis , we recently finalized a decision brief on next steps for this issue. We aligned upon revoking the exception and working with WDQS users to ensure their federated queries are compliant with the UA policy. To ensure we have enough lead time to communicate the upcoming change, we are requesting that the policy exception remain in place until February 18th, 2026.

Please let me know if this poses any issues on the SRE side.

I like the idea of having a full-stack view of traffic problems, especially on the media-serving side of things, which I'd focus on more at first. I would actually create two separate dashboards for text and media-serving.

Hi, I still see a lot of requests from your IPs with user-agent Faraday v2.14.0.

I'll tag the task as solved, please let us know if you still enocounter problems. Thanks to @Reedy for the assistance over the weekend!

In T414173#11509648, @Xqt wrote:

@Fabfur: I still have the 429 problem with my bot. Here the html content:

I have added an hotfix that should go live soon-ish (in the next hour or so).

I guess there's some bug in AutoWikiBrowser so that those requests lack authentication cookies; otherwise you wouldn't get that error, which can only happen for logged-out users.

In T414204#11507846, @Wikiwan wrote:

Hi Joe,
I have, and was suggested to open this ticket

@Wikiwan I guess you got some message with the 429 responses, right? And those messages didn't suggest to come open a task, but rather to contact an email address.

To clarify:

In T414173#11507634, @Xqt wrote:

@Joe, @Tgr: Could you please consider postponing the newly introduced restriction until the Pywikibot User-Agent has been updated? As far as I know, the current implementation breaks a large number of bots that rely on it. Postponing the restriction for a short period would allow us maintainers to fix the User-Agent without causing widespread disruptions. Thank you for your consideration.

This user agent is not compliat with our user-agent policy:

I'm not sure having your CI depend on external resources is a good policy; I encourage you to change that long-term, but anyways, we don't want to block the work on pywikibot right now.

Exception added. I allowed a generous amount of requests; please let us know if you still run into problems.

@Ragesoss I see you still get blocked from time to time; I will add an exception, per https://wikitech.wikimedia.org/wiki/Robot_policy#What_to_do_if_these_limits_are_too_strict_for_me?, using the IPs that you just provided.

@Ragesoss as far as I can tell, the problem is you are not honoring the wikimedia User-Agent policy, and we have recently started to enforce stricter rate limits for bots that dont' respect the policy, hence you are being blocked.

@Ragesoss what is the User-Agent you use when making those requests?

In T396807#11498251, @aaron wrote:

We don't need the sandbox to work on www.wikimedia.org, just wikimedia.org. We currently use www.wikimedia.org for rest_v1-wikimedia.json, but I've been thinking about copying that file to standard-docroot and pointing the MediaWiki REST sandbox to that file on meta.wikimedia.org (then deleting the old file later). Basically, I'd like to avoid www.wikimedia.org completely.

Indeed T391397 is the cause of that other rule. I think we should just ban their UA completely.

In fact, we have a requestctl rule https://requestctl.wikimedia.org/action/cache-text/datadog_synthetics for rate-limiting them. The rule is only applied to cache misses, because we had no reason to block cache hits.

Datadog synthetics is a service from Datadog that we at some point had to block. Apparently some people like to use Datadog's agentic monitoring to check our infrastructure, for unforseen reasons. I wouldn't be sure this is not something fishy though, I'll look a bit more into it.

@TheDJ the 429s are probably due to the massive scraping activity SREs have had to deal with over the festivities.

Can you please report the full error message you get in the response body? Thanks

A suggestion: when reporting the latencies, it would be useful to also have the standard deviation, which ab(1) provides, as the numbers seem so close to each other that the differences might well be within one standard deviation from each other between various solutions. That might help pick the one with the better size/performance tradeoffs better.

This problem is blocking KR work for WE5, responsible content reuse. Please treat the problem with some urgency, this code is preventing us from rate-limiting severely non-standard thumbnail size generation, which is one of the pillars of our work.

I'm very happy you're going with the option @jijiki recommended, which sounds like both the path of least resistance and the best option.

As my closing 2 cents about this whole system:

In T409253#11343878, @LSobanski wrote:

As an aside, this task doesn''t meet Unbreak Now criteria as defined in https://www.mediawiki.org/wiki/Phabricator/Project_management#Priority_levels, which are focused on MW train right now. This possibly warrants a conversation on whether the document needs an update.

cc @Aklapper