0

I am running a website for a customer who has terrabytes worth of images. Each JPG image is high-res (20MB) and belongs to a hierarchy like this:

Group A SubGroup 1 subsubgroup a Image A1a.1 Image A1a.2 ...

Currently the customer can only view images for any given subsubgroup at any one time, meaning only ~30 images get loaded on the page. Not a problem.

The images are stored in a cloud bucket (GCP). Currently, my server just sends the pre-signed URLs and the client loads them.

The customer has request the ability to mass download all images in a Group from their browser.

Some Groups can contain hundreds of SubGroups, and each one of these can contain multiple subsubgroups. As an illustrative example, for a Group of 20 SubGroups, we're talking about 20GB of data.

How can I achieve this reliably?

Possible solutions:

  1. Download all images on the server, create a zip (or number of zips) and upload to bucket. Share link with customer. Would probably need to run as a background job. This feels very flaky and I am doubtful it'll even work given the size of the data.

  2. Create a new page that renders all images. My server only sends the pre-signed URLs from GCP bucket. Add some basic JS functionality to download all images on the page.

  3. Some sort of SFTP server?

My current stack is relatively minimal:

  • 1 cloud VM
  • Flask app, uwsgi server, behind an NGINX reverse proxy
  • Cloud storage (GCP) for images
  • MySQL database for image hierarchies
Improve this question
6
  • "Create a new page that renders all images. My server only sends the pre-signed URLs from GCP bucket. Add some basic JS functionality to download all images on the page." You don't need to render them on the page in order to download them with JS. You just need the URLs.. Commented Sep 24, 2024 at 16:09
  • 1
    20Gb downloaded from a browser? Forget about it, that's an unrealistic requirement. SFTP won't help you here. What will you do when connection drops for whatever reason? Torrent protocol - that's what you need. Commented Sep 24, 2024 at 17:25
  • 2
    But honestly, this sounds like XY problem. Make thumbnails out of those images and serve them to users, paginated. No one looks at 1000 images at the same time. Serve full images only on demand. Commented Sep 24, 2024 at 17:27
  • maybe : medium.com/google-cloud/… Commented Sep 24, 2024 at 18:30
  • As freakish said, so the user can scroll through page after page of images without delay. I might add two levels of compressed images and only use the 20 MB for zooming in or for print. Commented Sep 24, 2024 at 21:05

3 Answers 3

Reset to default
3

If the requirement is really to download a huge number of images and no just scroll through them, then the ZIP solution would probably work well. I've recently had the need to download a huge number of vacation photos from my Google account, and they offer exactly this as a mass download option.

Creating the ZIP files should actually not be too time-consuming, as you would not need to compress the files, just copy them to the archive (JPEG images are well compressed already).

You need to manage your user's expectations, though. Downloading 4GB over a reliable broadband connection is pretty OK, 20GB over an ADSL link will likely not make them happy. If you're really generous, you can offer a set of ZIP files with some size limit so they can download the data piecewise.

Improve this answer
1

If the images are already stored in a GCS bucket, you should leverage the fact that cloud storage solutions are already very good at exactly what you want to do, which is transfer large amounts of data reliably.

If the customer needs to download a large number of huge files, use existing tools such as gsutil (GCS's command line tool) to download them. It should be hard to build the appropriate command line to recursively download a Group from GCS and present it to the customer, instructing them to run it locally.

Don't reinvent the wheel. GCS has a supported tool for reliable, resumable file downloads. Use it.

Improve this answer
1
  • I would 100% be using gsutil server side for this, but there's no way I can get the customer to run it locally. This is a non-technical customer, multiple users, and other restricitions which I haven't mentioned for the sake of simplicity. But thank you for the suggestion. Commented Sep 25, 2024 at 13:02
0

You might be overthinking this or, rather, creating a challenge that doesn't really exist, if you think about this a little differently.

I would suggest that you think about this not as a 'mass' download, but as a local, persistent disk-based cache of the images. Your groups of (sub)groups has the same structure as folders in a file system. The general idea here would be to create a local user directory which your interface creates and manages.

When the user views a group (or sub-group), your interface first checks the local path that matches your group/subgroup structure. If a file is found locally, you can display that. If it not, you pull from GCP.

The last part of this would be how the local disk store is populated with the actual images. You could provide options to save or cache individual files, or leaf groups upon viewing, or you could recursively save groups/subgroups. This avoids the challenges around downloading very large files because you are simply pulling the images individually. And really, there's not much advantage to zipping because the images won't compress significantly, and you basically need the full file to get anything out of it. If you fail to get an image, you only miss that one image and you can build a retry process for those failures, e.g.: re-add the URL to the end of the download list.

The biggest challenge here, I think, is dealing with local file access and directory management from a browser-based JS solution. Seems like there's been some improvement in the capabilities required for that kind of solution over time, but it's not really my area of expertise.

Improve this answer

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.