22

I was asked how to save all the magento media assets in Amazon S3. I thought about using a PHP Stream Wrapper like this one: https://github.com/punkave/aS3StreamWrapper and just set the media folder to s3:\mybucket\mediaFolder

Does anyone has experience with this?

Or better use something like this: https://github.com/sstoiana/magento-s3 But this extension doesn't look good. Any other recommendations?

Improve this question
5
  • 2
    Just as an aside. What is your motivation exactly? Unless you are catering to a global audience, or pushing more traffic than your server's interface (>100Mbit/1Gbit), or have a server provider with poor transit (which is an issue in itself) - CDN's usually are slower than serving the content direct from the same machine. Congested networks, distant break out points and poor routes usually lead to slower content delivery than simply hosting it yourself. Commented Feb 1, 2013 at 20:53
  • 1
    The motivation is: Someone asked me for my experience and opinion and I thought about it and prefer the stream wrapper solution, but we have this cool site and maybe someone has his two cents for me :-) Commented Feb 1, 2013 at 22:12
  • 1
    @sonassi your experience with CDNs seems to be somewhat dated. All the mature CDN solutions distribute content geographically at a much higher scale than you will be able to afford for your web servers and will serve content up from the closest geographical location to the requesting client. It's also important to optimize # of HTTP requests on your site by using an asset pipeline for js/css concatenation and minification, but that's much more complicated especially for Magento sites. Commented May 24, 2013 at 17:24
  • @RalphTice Not that outdated considering we run our own BGP anycast network across 3 continents. Most 'affordable' CDNs have single PoPs per country (some just per continent), that isn't necessarily geographically any closer than your web server itself. Eg. GB's nearest PoP for Amazon is Ireland - which has far, far greater latency than a England based machine. So I reiterate, that unless you are targeting a global audience, its unlikely your domestic customers would see any benefit at all from a CDN. In some cases it is slower, not to mention more expensive and complicated. Commented May 24, 2013 at 22:04
  • guys, i have used thaiphan/magento-s3 extension for s3 bucket, its working with upload product images on s3 and connect with product images , but it will also generate product images in my system server also. i just need to use s3 bucket for images so i also tried to delete my system's pub/media/catalog but after load site page product images is recreated in my system server also. is there need to stop this image created in local server ? Commented Apr 19, 2018 at 12:09

6 Answers 6

Reset to default
7

it seems these guys find a solution: ( http://thinkglobal.co/resources/moving-the-magento-media-directory-to-s3/ )

There are 2 possible approaches, one being to have Magento push all media assets to S3 as they are created and the other being to simply mount the media directory to an S3 bucket and let Magento think that it’s writing to the local disk. We chose the latter.

We ended up using a tool called s3fs, which uses a broader technology called fuse. With some fairly straightforward configuration (described below), you can get s3fs up and running. Be careful not to hardcode S3 bucket names or API access credentials if you are using AWS or other PAAS provider.

Make sure you replace the following variables with values for your specific configuration:

  • {{S3_BUCKET}} (name of your S3 bucket)
  • {{API_PUBLIC_ACCESS_KEY}}(provided by AWS)
  • {{API_SECRET_ACCESS_KEY}} (provided by AWS)
  • {{UID}}(user id for nginx/apache user)
  • {{GID}} (group id for nginx/apache user)
  • {{MOUNTED_DIRECTORY}} (path to your magento media directory)

Here is the setup:

yum install -y gcc libstdc++-devel gcc-c++ curl curl* curl-devel libxml2 libxml2* libxml2-devel openssl-devel mailcap cd /usr/local/src wget http://downloads.sourceforge.net/project/fuse/fuse-2.X/2.9.3/fuse-2.9.3.tar.gz tar -xzf fuse-2.9.3.tar.gz rm -f fuse-2.9.3.tar.gz mv fuse-2.9.3 fuse cd fuse/ ./configure –prefix=/usr make make install export PKG_CONFIG_PATH=/usr/lib/pkgconfig:/usr/lib64/pkgconfig/ ldconfig modprobe fuse pkg-config –modversion fuse wget http://s3fs.googlecode.com/files/s3fs-1.74.tar.gz tar -xzvf s3fs-1.74.tar.gz rm -f s3fs-1.74.tar.gz mv s3fs-1.74 s3fs cd s3fs ./configure –prefix=/usr make make install echo “{{S3_BUCKET}}:{{API_PUBLIC_ACCESS_KEY}}:{{API_SECRET_ACCESS_KEY}}” > ~/.passwd-s3fs chmod 600 ~/.passwd-s3fs echo “user_allow_other” > /etc/fuse.conf echo “s3fs#{{S3_BUCKET}} {{MOUNTED_DIRECTORY}} fuse use_cache=/tmp,allow_other,uid={{UID}},gid={{GID}} 0 0″ >> /etc/fstab s3fs -o allow_other -o uid={{UID}} -o gid={{GID}} -ouse_cache=/tmp {{S3_BUCKET}} {{MOUNTED_DIRECTORY}}
Improve this answer
3
2

OnePica's CDN extension is very handy out of the box for this functionality and supports more than just S3. Quote: "Currently, the extension integrates with Amazon S3, Coral CDN, Mosso/Rackspace Cloud Files, and any CDN, server or service that supports FTP, FTPS, or SFTP."

Improve this answer
7
  • 5
    OnePica is only CDN manager, means the files are still local Commented Feb 4, 2013 at 20:01
  • @FabianBlechschmidt You'll have to be more specific with your comment. OnePica's extension most certainly does not leave the files locally when properly installed/configured and I'm using it in production myself w/ Amazon S3. Commented Feb 6, 2013 at 15:51
  • Maybe I missed something. You mean the files are uploaded to S3 and no more in media/catalog/... ? Commented Feb 6, 2013 at 19:15
  • I believe they still get saved in /media/catalog/... but they're still uploaded to S3 and get served from S3. Commented Feb 6, 2013 at 20:42
  • 1
    FYI, this should also work with DreamObjects drop-in S3 replacement solution. It has worked great with every other S3 bucket so far. Commented Apr 24, 2013 at 16:53
1

OnePica's CDN extension is not the solution it only works for product images not to mention you need to set the correct TTL for its own cache life time which will timeout much earlier than default Magento. All you cms wysiwyg image will be in media folder but not using S3, you will end up uploading all your media image to S3.

Use cloudfront will be good but you will still need a NFS to share your media folder across all your frontend nodes and backend nodes. Or you need to build your own interface to save file on S3, and you also need to handle errors (think about someone upload extreme large file whether your code can handle that before S3 API timeout)

Other way just save media storage in your extra RDS (database) and individual resize cache image will still save in your app server local media folder, but it is ok as cloudfront will cache them already.

Improve this answer
1

I prefer having a separate "admin" node(admin.example.com) and keeping media folders there. This way you would save yourself with NFS cost as well as issue with syncing etc. any CDN (I use cloudfront) could then be connected with admin node media folders, and be configured for all stores.

Other benefits of using admin node is, you could move all crons(reindexing and other crons) , scheduled tasks(product imports, inventory & order sync etc) at admin node and save frontend nodes cpu processing.

Edit: As last answer didnt actually addressed OP's question. Magento Media Assets in Amazon S3? Well I believe Magento already has got so much processing overhead, that I think it'd be good idea to keep it from s3 syncing/streaming, and use local media folder instead.

Improve this answer
1
  • Thanks! Actually I checked out a cloud host where no upload was possible. Dropped it later :-) Commented Oct 28, 2014 at 8:26
1

Well, now that a couple years have gone by I think it's important to note that there are multiple plugins available that will move and serve assets from S3 / CDN.

Google searches turn up plenty of options.

Here's one on GitHub:

https://github.com/magefm/cdn

Here's one on Magento's Connect marketplace:

http://www.magentocommerce.com/magento-connect/aws-s3-cdn-for-product-images.html

I have not tried these particular extensions yet but will be for an upcoming project. There are others out there, too.

I'd have to disagree with @Sonassi, using separate storage to allow the cheapest server configuration still with good performance is really considered the best practice today. For example, simply use S3 to move all static content off your web server helps the server use far less bandwidth and allow more concurrent users and less overall server power needed. And, if you're in a single datacenter like most people these days, and say it's in Dallas, then there's no doubt that people in Western Europe will be served much faster with a Global CDN like Cloudfront. So, using storage separate from your web server to lower bandwidth usage allows you to use only 1 server when you would need more sooner without this approach, and using CDN will greatly increase global load times.

Improve this answer
0

Found another extension built by arkadedigital https://github.com/arkadedigital/magento-s3

Improve this answer

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.