Questions tagged [amazon-s3]
Amazon Simple Storage Service. For questions about storing and retrieving DevOps-related data in S3, or using S3 for building a code pipeline.
63 questions
0 votes
0 answers
30 views
Kubernetes vs Object Storage for static website
I'm evaluating the best approach for serving the static assets (JavaScript bundles, CSS, HTML, etc.) of a React application that uses Module Federation and Webpack. I have two potential options: ...
0 votes
1 answer
92 views
Azure DevOps release to AWS fails with bucket does not exist
We are building a simple website and pushing it to an S3 bucket on AWS with a release pipeline. We configured the service connection in ADO with an access key and secret from AWS and the deploy was ...
- 101
0 votes
1 answer
91 views
CD with CodeDeploy and GitHub Actions - why specify S3 bucket?
Following tutorials like this one, I managed to set up a continuous deployment pipe to deploy automatically via AWS CodeDeploy to my EC2 instance using GitHub actions, triggered on merging (or rather ...
- 101
0 votes
1 answer
99 views
How to design AWS infrastructure for my online course selling website?
I am designing an AWS architecture for hosting a website that sells online courses and merchandise products. For the purpose, I am planning to use the following architecture design: 1 VPC 1 ...
- 27
0 votes
1 answer
63 views
How to ascertain excesive use of my S3 bucket?
I have a S3 bucket that I use to store images for a website I manage. The bill is usually about $1/month and I store about 12.5 GB on the service. It's never really been an issue before, but recently ...
4 votes
4 answers
5k views
How can I use S3 as a filesystem
I'm not sure if it's something I'm doing wrong, or if the S3 web interface is just trash. There are a ton of basic capabilities that work in pretty much any other file storage product (Windows ...
- 172
0 votes
1 answer
1k views
s3: reading parquet files with a subset of columns as argument; does this reduce data transfer?
I have set of large tables stored as parquets on s3. In python, I'm using: pd.read_parquet(...,columns=columns) I'm reading the files directly from s3, without any database engine whatsoever for ...
0 votes
1 answer
2k views
calculating the size of objects in AWS S3 buckets
I ran into an problem recently, my AWS S3 bucket shows total size of 30 TB. However when i try to get the total size of individual folders, it is not summing up more than 2 TB. I used the below ...
- 3
0 votes
1 answer
423 views
When OnPrem with Kubernetes, what is the recommended way to do file storage buckets?
When you are running kubernetes on GKS, AKS, etc, you use your cloud provider specific storage bucket such as Amazon S3, google cloud storage, azure blob storage, to store your raw binary content in a ...
2 votes
1 answer
646 views
How to assign an ACL for each S3 Bucket in a tuple/list using Terraform?
I have a terraform file set to create a list of S3 buckets, and I was assigning acl to each bucket the old way: resource "aws_s3_bucket" "this" { count = length(var....
- 41
3 votes
1 answer
1k views
Use S3 To Store ConfigMap
I have a ConfigMap which stores Nginx configs and It is getting size exceeded. There are many site configs in there so I can't create multiple ConfigMaps. I thought of using S3 to store ConfigMap, is ...
- 133
2 votes
1 answer
1k views
Terraform Aws S3 - deny all users except for a specific user
I have a bucket which I need to restrict to a specific user, I have written the following script but it still seems to allow all users to operate on the bucket. resource "aws_s3_bucket" &...
- 121
0 votes
1 answer
3k views
How can I efficiently scale a data lake?
I am running into problems with AWS Athena data lakes. They were initially very quick, but as we've scaled, they're performing poorly and are becoming expensive and inefficient. What can we do to ...
- 45
0 votes
1 answer
154 views
How can I grant access to my cloud storage for clients?
I am developing a webshop for large digital products (multiple .tif image files). These are currently stored in an S3 bucket. What is a recommended way to grant access to my customers once they have ...
- 11
1 vote
1 answer
602 views
What is the proper AWS IAM policy for access to a single bucket that allows stacking?
I am working on a buildout where I am doing the following. Eventually I will need to make this scalable so I am not doing this manually each time. I have several clients dumping data into S3 buckets. ...
- 131