Backbeat is an engine with a messaging system at its heart. It's part of Zenko, Scality’s Open Source Multi-Cloud Data Controller. Learn more about Zenko at Zenko.io
Backbeat is optimized for queuing metadata updates and dispatching work to long-running tasks in the background. The core engine can be extended for many use cases, which are called extensions, as listed below.
This feature replicates objects from one S3 bucket to another S3 bucket in a different geographical region. The extension uses the local Metadata journal as the source of truth and replicates object updates in a FIFO order.
- Backbeat core design
- CRR to AWS S3 workflow
- Object Lifecycle management
- Metrics
- CRR Pause and Resume
- Healthcheck
- Site Level CRR
- Transient Source
- Out of Band updates from RING
This guide assumes the following:
- Using MacOS
brewis installed (get it here)nodeis installed (version 6.9.5)yarnis installed (version 3.10.10)awsis installed (version 1.11.1)
mkdir ~/kafka && \ cd ~/kafka && \ curl https://archive.apache.org/dist/kafka/0.11.0.0/kafka_2.11-0.11.0.0.tgz | \ tar xvz && \ sed 's/zookeeper.connect=.*/zookeeper.connect=localhost:2181\/backbeat/' \ kafka_2.11-0.11.0.0/config/server.properties > \ kafka_2.11-0.11.0.0/config/server.properties.backbeat Start the zookeeper server:
zookeeper-server-start kafka_2.11-0.11.0.0/config/zookeeper.properties In a different Shell start the kafka server:
kafka-server-start kafka_2.11-0.11.0.0/config/server.properties.backbeat The "Failed Entry Processor" (started by Queue Populator) stores the "failed replication entries" to the Redis sorted set.
zkCli -server localhost:2181/backbeat create /queue-populator kafka-topics --create \ --zookeeper localhost:2181/backbeat \ --replication-factor 1 \ --partitions 1 \ --topic backbeat-replication kafka-topics --create \ --zookeeper localhost:2181/backbeat \ --replication-factor 1 \ --partitions 1 \ --topic backbeat-replication-status kafka-topics --create \ --zookeeper localhost:2181/backbeat \ --replication-factor 1 \ --partitions 1 \ --topic backbeat-replication-failed git clone https://github.com/scality/Vault ~/replication/vault && \ cd ~/replication/vault && \ yarn i Run vault with "in-memory backend"
chmod 400 ./tests/utils/keyfile VAULT_DB_BACKEND="MEMORY" yarn start
⚠️ with "in-memory backend", all data is lost after you stop the process.
or with a "mongodb backend"
chmod 400 ./tests/utils/keyfile VAULT_DB_BACKEND="MONGODB" yarn start mongodb can be installed follwing these steps
git clone https://github.com/scality/cloudserver ~/replication/cloudserver && \ cd ~/replication/cloudserver && \ yarn i Generate AWS access/secret key with full IAM and S3 access.
Add your keys aws configure --profile aws-account
Create AWS destination versioning-enabled bucket.
aws s3api create-bucket --bucket <DESTINATION_BUCKET_NAME> --profile aws-account aws s3api put-bucket-versioning \ --bucket <DESTINATION_BUCKET_NAME> \ --versioning-configuration Status=Enabled \ --profile aws-accountReplace existing ./locationConfig.json with:
{ "us-east-1": { "details": { "supportsVersioning": true }, "isTransient": false, "legacyAwsBehavior": false, "objectId": "0b1d9226-a694-11eb-bc21-baec55d199cd", "type": "file" }, "aws-location": { "type": "aws_s3", "legacyAwsBehavior": true, "details": { "awsEndpoint": "s3.amazonaws.com", "bucketName": "<DESTINATION_BUCKET_NAME>", "bucketMatch": false, "credentialsProfile": "aws-account", "serverSideEncryption": true } } }Update ./config.json with
"replicationEndpoints": [{ "site": "aws-location", "type": "aws_s3" }],In ./config.json, make sure recordLog.enabled is set to true
"recordLog": { "enabled": true, ... } Run Cloudserver
S3DATA=multiple S3METADATA=mongodb REMOTE_MANAGEMENT_DISABLE=true \ S3VAULT=scality yarn start Create a "Zenko" account and generate keys
bin/vaultclient create-account --name bart --email dev@backbeat --port 8600 bin/vaultclient generate-account-access-key --name bart --port 8600 aws configure --profile bart git clone https://github.com/scality/backbeat ~/replication/backbeat && \ cd ~/replication/backbeat && \ yarn i Update conf/authdata.json with bart informations and keys.
{ "accounts": [{ "name": "bart", "arn": "aws:iam::331457510670:/bart", "canonicalID": "2083781e15384e30f48c651a948ec2dc1e1801c4af24c2750a166823e28ca570", "displayName": "bart", "keys": { "access": "20TNCD06HOCSLQSABFZP", "secret": "1P43SL0ekJjXnQvliV0KgMibZ=N2lKZO4dpnWzbF" } } ] }Update conf/config.json section extensions.replication.source.auth
"auth": { "type": "account", "account": "bart", "vault": { "host": "127.0.0.1", "port": 8500, "adminPort": 8600 } }Make sure conf/config.json section extensions.replication.destination.bootstrapList includes:
{ "site": "aws-location", "type": "aws_s3" }S3_REPLICATION_METRICS_PROBE=true REMOTE_MANAGEMENT_DISABLE=true \ yarn run queue_populator S3_REPLICATION_METRICS_PROBE=true REMOTE_MANAGEMENT_DISABLE=true \ yarn run queue_processor aws-location S3_REPLICATION_METRICS_PROBE=true REMOTE_MANAGEMENT_DISABLE=true \ yarn run replication_status_processor Create a source bucket with versioning enabled:
aws s3api create-bucket \ --bucket sourcebucket \ --endpoint-url http://127.0.0.1:8000 \ --profile bart aws s3api put-bucket-versioning \ --bucket sourcebucket \ --versioning-configuration Status=Enabled \ --endpoint-url=http://127.0.0.1:8000 \ --profile bart Create replication.json
{ "Role": "arn:aws:iam::root:role/s3-replication-role", "Rules": [ { "Status": "Enabled", "Prefix": "", "Destination": { "Bucket": "arn:aws:s3:::sourcebucket", "StorageClass": "aws-location" } } ] }aws s3api put-bucket-replication \ --bucket sourcebucket \ --replication-configuration file://replication.json \ --endpoint-url=http://127.0.0.1:8000 \ --profile bart aws s3api put-object \ --key key0 \ --body file \ --bucket sourcebucket \ --endpoint-url=http://127.0.0.1:8000 \ --profile bart aws s3api head-object \ --key bucketsource/key0 \ --bucket <DESTINATION_BUCKET_NAME> \ --profile aws-account In our $HOME directory, we now have the following directories:
$HOME ├── kafka │ └── kafka_2.11-0.11.0.0 ├── replication ├── backbeat ├── cloudserver └── vault