40

Starting around a couple of hours ago, attempts to upload images via the image uploader onto SE's image servers have been failing.

This has been confirmed across multiple SE sites (so it's not SE site specific) but doesn't affect uploads in SE chat (thanks to those SE members who kindly did those checks).

Repro: With a known valid (e.g. small enough ) image, in the image upload dialog, people can browse to the local file and the local thumbnail shows in their browser. All good so far. However, clicking the "Add picture" button causes a pause of a few seconds and then the error message in red appears:

An error occurred, please try again.

Here's a screenshot:

screenshot of error message after attempting to upload image to SE image servers

Please can SE staff fix the problem?

Improve this question
12
  • 10
    Can replicate the issue on my end Commented Jun 7 at 0:33
  • 11
    Able to reproduce on the sites, but it works on chat. On chat, image uploads are quite slow but work fine. Most likely it's taking too long, resulting in a timeout on the sites, but chat sites have a larger or nonexistent timeout period. Commented Jun 7 at 0:34
  • 7
    Also able to reproduce -- main & meta site image uploads fail while chat uploads work. Commented Jun 7 at 0:45
  • 2
    @SonictheAnonymousHedgehog Yes, chat upload takes a very long time, but does work. On the main sites image upload fails with an unknown error after only about 21 seconds or so. That seems somewhat for a timeout, at least taken at face value. Chat uploads seem to take longer than that. Perhaps there could be component protocol failures at a finer granularity than is easily obvious to a casual observer. Commented Jun 7 at 2:37
  • 1
    This is probably also causing not being able to change profile pictures (or maybe it’s just a me problem?) Commented Jun 7 at 4:04
  • 3
    One of the profile picture sources supported is SE image hosting so that might track Commented Jun 7 at 4:12
  • 20
    Can confirm - we appear to be having a (possibly intermittent) issue with a third-party service we use for the security scanning of the uploads. I'm not familiar with this part of our system, so I unfortunately can't offer much more, but I've escalated this internally and we'll get folks on it tomorrow (or, fingers crossed, the issue will resolve itself by then). Thanks for the report! Commented Jun 7 at 5:12
  • 2
    It looks like the issue has been fixed, or has resolved itself. Commented Jun 7 at 16:50
  • 2
    @PM2Ring The latter, although I was just about to deploy a mitigation. I'm choosing to believe that intimidated everything into getting back to work. Said mitigation is deployed anyway, so if this happens again, we should be in a better state. Commented Jun 7 at 17:44
  • 2
    @PM2Ring - Hi, Thanks - as you said, image upload seems to be working again which is good news. However, until we get a statement from SE staff that something has actually been fixed, then we don't know if whatever-caused-it could fail again. From here: "If you didn't fix it, then it ain't fixed" :) || Let's hope for an update from SE staff when they are confident of a fix, along with an explanation of the cause for the records :) Commented Jun 7 at 17:44
  • @AdamLear - Hi, Our comments crossed :) If you're confident of the fix, will you be posting an answer, or are you waiting to test the mitigation, or ...? Thanks for your help with this. Commented Jun 7 at 17:45
  • 5
    @SamGibson I posted an answer. I'm going to keep an eye on things over the weekend, but I don't really expect new problems here. Commented Jun 7 at 17:53

3 Answers 3

Reset to default
26

A third-party service we use for the security scanning of the image uploads was having issues. Basically, we were seeing increased latency on our requests and that caused the uploads to time out.

The issue has resolved itself from our perspective, but we also deployed an increased timeout value so that if this happens again, we should hopefully continue to allow uploads. I'm optimistic about it since that's why chat uploads were still working - the chat application ran with a higher allowed timeout.

Improve this answer
6
  • Did the service fix itself or did you change to a different service? Commented Jun 7 at 17:54
  • 1
    @bobble It fixed itself. Commented Jun 7 at 17:55
  • 1
    @AdamLear - Thanks again for your help with this. Increasing the timeout (and an apparent reduction in latency from the service provider just before you kindly increased the timeout) seems to have resolved things this time. (I plan to accept the answer after the weekend, if all goes well). || However using a "think beyond the fix" troubleshooting approach, can we do even better? Can SRE (or equivalent function in SE engineering) add a monitoring task which proactively measures the latency of that scanning service, to trigger an alert? What if the latency suddenly exceeds even the new timeout? Commented Jun 7 at 18:02
  • 6
    @SamGibson Yes, of course. There will be a few follow-ups from this, but none of that needs to happen on a weekend. Commented Jun 7 at 18:10
  • @AdamLear - Thanks, I just wanted to understand if those follow-ups are planned. I didn't say anything about them needing to be done at the weekend :) From experience, I know that when the urgency around a problem isn't there (i.e. the problem has gone away), sometimes the appetite for RCA can disappear too. || Thanks again for your help, especially at the weekend. If everything still seems to be working early next week, I plan to accept the answer in the knowledge that the follow-ups will happen in the background :) Is that plan OK with you? Commented Jun 7 at 18:19
  • 4
    @SamGibson Works for me. And thanks again for reporting the problem! Commented Jun 7 at 18:21
11

A workaround is to upload the image in chat, eg a Sandbox room (only available to users with at least 100 rep) , and then upload that image into your question or answer using the normal image uploader. It will "upload" instantly because it's already on the server.

Here's an example I just tested a few minutes ago.

Oklch hue cycle

Once you have the URL, you can delete the image in chat, to avoid creating clutter.

Update

It looks like normal image uploading is working again.

Test:
Small hue cycle

Improve this answer
16
  • 1
    Uploading images that way will automatically post them into the chat room, which can lead to unnecessary spam. I just tested it — as soon as you upload an image, it appears instantly in the chat. I don’t think we want to clutter the room like that. Commented Jun 7 at 5:25
  • @HarryMcKenzie That's why I said "you can delete the image in chat, to avoid creating clutter". Commented Jun 7 at 5:30
  • Yes, if followed strictly, it works, but you'll still see a "(removed)" spam entry. Also many users may not go back to delete it, which can lead to clutter over time. It's probably best to wait until it is fixed. Commented Jun 7 at 5:33
  • 1
    sandbox works fine for that. Else maybe a designated temporary image upload room? Commented Jun 7 at 5:38
  • @Harry The (removed) entries aren't really an issue, and they aren't visible in the chat transcript. And if you use one of the Sandbox chatrooms, clutter isn't much of an issue. There's a lot of clutter there already. ;) Commented Jun 7 at 5:40
  • 1
    Cool, then I guess there’s no real issue for now, haha. Still, I think Journeyman Geek’s idea of a temporary image upload room is actually cool workaround too! :) Commented Jun 7 at 5:52
  • 2
    @PM2Ring - Thanks for this suggestion. Without me creating a brand new account to test, do you know if the usual 20 point requirement applies before users are able to post in the sandbox chatroom? If it does, then that would prevent new users (e.g. new OPs) joining that chatroom to upload images there. Perhaps this would be a reason to point new OPs to the "Stack Exchange Lobby" chatroom instead, since that chatroom doesn't have the 20 point requirement AFAIK. Thoughts? Commented Jun 7 at 5:55
  • 2
    @SamGibson good point. It only works for 20 points reputation so new users cannot use this workaround. I just tested it. Commented Jun 7 at 6:01
  • 1
    @SamGibson I think you may need even more rep points (100 ?) to upload in chat. The new Lobby rooms originally permitted anyone to post image links, but that has been changed, because some porn images & anims were posted. Commented Jun 7 at 6:08
  • @PM2Ring - I see, thanks. So uploading via chat may be an option for some people, but not for all. || If anyone definitely knows the points/rep restrictions for uploading images via chat (normal chatrooms & lobby rooms, if different), please do leave a comment. Commented Jun 7 at 6:12
  • @SamGibson i.sstatic.net/gwooqf0I.jpg Commented Jun 7 at 6:17
  • 1
    @Sam "At 100 reputation you also get access to the "Upload image" button in chat". meta.stackexchange.com/a/317856/334566 Commented Jun 7 at 6:17
  • @Harry 20 points is the rep needed to talk in chat. Low rep users can normally upload images on the main (& meta) sites, but they need at least 10 points to make any image link visible, as an anti-spam measure. Commented Jun 7 at 6:21
  • 1
    @Sam Sorry for the confusion. On the main sites, anyone can (normally) upload images, or link to externally hosted images (eg on Wikipedia) but those images will not be displayed as images if they have <10 rep, you just see the link URL. But that has nothing to do with chat. Commented Jun 7 at 6:34
  • 2
    @PM2Ring - Got it, thanks :) AIUI everything still points to a 100 rep requirement to upload images via chat. So that will be a useful option for some people (e.g. active members who already have that rep), but not for new OPs. Commented Jun 7 at 6:42
7

While the following workaround does function temporarily for users with less than 100 reputation, as noted in this post, it's best not to rely on it long-term. If you have at least 100 reputation, consider using the chat-based image upload workaround instead.

That's a very important point. This issue really needs to be addressed as soon as possible — users, especially those with less than 100 rep, are already turning to external image hosts like Imgur just to get their questions posted. If this continues, we'll likely end up with a large number of posts missing images in the future once those external links break or expire. Time is of the essence if we want to preserve the quality and integrity of content across the Stack Exchange network.

Confirmed: this issue affects all Stack Exchange sites. In the meantime, here’s a simple workaround for those who aren’t aware: Manually upload your image via imgur.com/upload. After uploading, right-click the image and select Open image in new tab

screenshot

Copy the direct image URL, in my case:

https://i.imgur.com/4ei46kq.png

Edit your question and embed the image using the following Markdown format (as shown in the red box):

[![screenshot][1]][1] [1]: https://i.imgur.com/4ei46kq.png

screenshot

Improve this answer
7
  • 4
    these would ideally need to be fixed later. Maybe a meta post to keep track? Commented Jun 7 at 4:09
  • 4
    Harry McKenzie - Thanks for this. I already wrote a (less good!) version for "my" SE site here. One concern using Imgur which I mentioned at that link (despite it being one of the least bad options) is that I remember during the transition to SE's own image servers, someone said that anonymous uploads to Imgur were going to have a short life before being removed. I can't find that post now :( My point is, as Journeyman Geek has just commented, anything uploaded to Imgur should ideally be copied onto SE's own servers when they're fixed. Commented Jun 7 at 4:19
  • 2
    @SamGibson That's a very good and important point. This issue really needs to be addressed as soon as possible, users are already turning to external image hosts like Imgur just to get their questions posted. If this continues, we'll likely end up with a large number of posts missing images in the future once those external links break or expire. Time is of the essence if we want to preserve the quality and integrity of content across the Stack Exchange network. Commented Jun 7 at 4:34
  • 3
    @HarryMcKenzie - Thanks Harry, I see we're thinking along the same lines. I expect your site of Blender.SE uses lots of images too! So thanks again for pushing this workaround to a wider audience (I should have thought to do that) as well as mentioning that concern about not using it for any longer than we have to. Commented Jun 7 at 4:48
  • 3
    I don't understand why you want to upload to imgur, rather than to upload via chat. Commented Jun 7 at 5:11
  • 3
    Because you can't upload to chat with reputation below 100. So this is only for users that have reputation below 100, which is required to upload to chat. Commented Jun 7 at 10:55
  • 1
    You can also use Copy image address instead of Open image in new tab > copying the URL from the address bar. Commented Jun 7 at 16:17

You must log in to answer this question.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.