A bosh release for deploying fluentd.
This release has been designed specifically for the usecase of shipping logs from syslog to S3. It should be fairly easy to adapt it for other usecases by adding more plugins in the future though.
The Concourse pipeline updates the blobs from the Gemfile and vendors new Ruby versions. To include a new version of fluentd or a dependency, change the version in fluentd.Gemfile and run bundle install --gemfile fluentd.Gemfile to recreate the lock file.
Job templates have some spec tests in spec. New job properties and/or template files should have new spec tests added.
To build a dev release locally, run:
bosh create-release --force # or with tarball bosh create-release --force --tarball fluentd-boshrelease.tgzFinal releases are built from the master branch and uploaded automatically by the pipeline. Changes that should be built into a final release should (ideally) be merged into master first and released via this method. If you have changes that you require a final release for that are not suitable for the master branch, a final release can be built locally with:
bosh create-release --final --tarball fluentd-boshrelease.tgzreleases: - name: fluentd version: 0.0.13 url: https://github.com/EngineerBetter/fluentd-boshrelease/releases/download/0.0.13/fluentd-final-release-0.0.13.tgz sha1: affd49680cdc99a5a158d68bda63cd6547939acf - name: "bpm" version: "1.1.13" url: "https://bosh.io/d/github.com/cloudfoundry/bpm-release?v=1.1.13" sha1: "82322898b2393951108617caac43752e498632a2" stemcells: - alias: default os: ubuntu-bionic version: "1.22" instance_groups: - name: fluentd stemcell: default vm_type: small networks: - name: default azs: [z1] instances: 1 jobs: - name: bpm release: bpm - name: fluentd release: fluentd properties: fluent: conf: | <source> @type syslog port 5140 bind 0.0.0.0 tag concourse <transport tcp> </transport> <parse> message_format rfc5424 </parse> </source> <label @FLUENT_LOG> <match fluent.*> @type stdout </match> </label> <match **> @type s3 s3_bucket $SOME_BUCKET s3_region eu-west-1 path concourse/%Y-%m-%d/ include_time_key true <buffer tag,time> @type file path /var/vcap/data/fluentd/tmp/s3-buffer timekey 30m timekey_wait 5m chunk_limit_size 64m flush_at_shutdown true total_limit_size 256m overflow_action block </buffer> <format> @type json </format> <instance_profile_credentials> ip_address 169.254.169.254 port 80 </instance_profile_credentials> </match> update: canaries: 1 max_in_flight: 10 canary_watch_time: 1000-30000 update_watch_time: 1000-30000 initial_deploy_az_update_strategy: serialYou can configure tls by adding the certificates to the properties section
properties: cert: ca: | -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE----- -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE----- crt: | -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE----- key: | -----BEGIN PRIVATE KEY----- ... -----END PRIVATE KEY-----and configure the path of the certificates as described below:
<transport tls> version TLSv1_2 ciphers ALL:!aNULL:!eNULL:!SSLv2 insecure false # For Cert signed by public CA ca_path /var/vcap/jobs/fluentd/certs/ca.crt cert_path /var/vcap/jobs/fluentd/certs/cert.crt private_key_path /var/vcap/jobs/fluentd/certs/cert.key client_cert_auth false </transport>You can run the tests with bundle:
bundle install bundle exec rspec spec/