Downloaded 39 times
Uploaded byDatabricks
Scaling and Modernizing Data Platform with Databricks
This document summarizes Atlassian's adoption of Databricks to manage their growing data pipelines and platforms. It discusses the challenges they faced with their previous architecture around development time, collaboration, and costs. With Databricks, Atlassian was able to build scalable data pipelines using notebooks and connectors, orchestrate workflows with Airflow, and provide self-service analytics and machine learning to teams while reducing infrastructure costs and data engineering dependencies. The key benefits included reduced development time by 30%, decreased infrastructure costs by 60%, and increased adoption of Databricks and self-service across teams.
In this document
Powered by AI
Presenters Esha Shah and Richa Singhal introduce managing data pipelines using Databricks and outline the agenda including Atlassian overview and challenges encountered.
Atlassian's data has grown 20x over the last 5 years, with 5x more internal users and handling billions of events daily.
Before Databricks, Atlassian faced challenges with development, team dependencies, cluster management, and collaboration.
Key strategies for scaling included self-service, standardization, automation, agility, and cost optimization.
Overview of current data architecture and success metrics: reduced development time, increased efficiency, and support for growth at lower infrastructure costs.
Discussion on building data pipelines using Databricks including orchestration and different methods like Databricks Notebooks and DB-Connect.
Utilization of Databricks Notebooks and DB-Connect, including multi-stage environments for release pipelines.
Focus on orchestration using Airflow on Kubernetes for resource tracking, job metadata, and quality management.
Utilization of Databricks Delta for advanced data handling features such as time travel and auto optimization.
Application of Databricks for analytics including exploratory analysis, strategy decisions, and collaborative self-service analytics.
Utilization in data science encompassing model training, scoring, experiment analysis, and increased ML adoption.
Summary with key takeaways: 30% reduction in delivery time, 60% decrease in infrastructure costs, and significant company-wide adoption of Databricks.
Thanking attendees followed by a request for feedback on the session.
Scaling and Modernizing Data Platform with Databricks
- 1. Managing & ScalingData Pipelines with Databricks Esha Shah Senior Data Engineer ATLASSIAN Go-To-Market Data Engineering Richa Singhal Senior Data Engineer
- 2. Agenda Atlassian Overview Summary Adopting Databricks DataPlatform Challenges
- 4. Growth over thelast 5 years Data is now 20x times (Multi petabytes) 5x growth in numbers of internal users 5x number of events/day (Billions)
- 5. Atlassian Data Architecture(Before Databricks)
- 6. Key Challenges withLegacy Architecture Development Cross-team dependencies Cluster management Collaboration
- 7.
- 8. Current Atlassian DataArchitecture
- 9. Our Success Story Reduceddevelopment time Rapid Development Increased team and project efficiency with simplified sharing and co-authoring Collaboration Were able to support growth while reducing Infrastructure cost Scaling Removed Data engineering dependency for Analytics and Data Science teams Self Service
- 10. Adopting Databricks atAtlassian Building Data Pipelines Orchestration Leveraging Databricks Delta Databricks for Analytics and Data Science
- 11.
- 12. Data Pipelines withDatabricks Data Pipelines using Notebooks Data Pipelines using DB-Connect
- 13. Development using DatabricksNotebook AWS Cloud Interactive Cluster Ephemeral Cluster Bitbucket Branch Databricks Workspace Import/ Export Jira Ticket Command Line Databricks Notebook Databricks Cluster
- 14. Multi-stage Envs usingDatabricks Workspaces Databricks Notebook Databricks Workspace Dev Folder Local/ Development Stage/ Production Bitbucket CICD Pipeline Stg Folder Prod Folder Stg Cluster Prod Cluster
- 15. Bitbucket CICD Pipeline branches: main: - step: name:Check configuration file deployment: test script: - pip install -r requirements.txt - 'yamllint -d "{extends: default, rules: {}" config.yaml' - python databricks_cicd/check_duplicates.py - step: name: Move code to Databricks deployment: production caches: - pip script: - pip install -r requirements.txt - bash databricks_cicd/move_code_to_databricks.sh prod - step: name: Update the job in Databricks script: - pip install -r requirements.txt - python databricks_cicd/configure_job_in_databricks.py
- 16. Development using DB-ConnectLibrary AWS Cloud Interactive Cluster Ephemeral Cluster Bitbucket Branch Local IDE Pull Request /Merge db-connect Jira Ticket Databricks Cluster
- 17. Multi-stage Envs usingAWS S3 Local IDE Databricks Cluster Dev Bucket Local/ Development Stage/ Production Bitbucket CICD Pipeline Docker Stg Bucket Prod Bucket Stg Cluster Prod Cluster
- 18.
- 19. Orchestration using Airflow Airflowon Kubernetes SparkSubmit Task YODA In-house Data Quality Platform SignalFx Opsgenie On-Call Notebook Task Slack Notification Code on S3 Notebook Databricks Workspace
- 20. Tracking Resource Usageand Cost Job Metadata 'custom_tags': { 'business_unit': 'Data Engineering', 'environment': cluster_env, 'pipeline': 'Team_name', 'user': 'airflow', 'resource_owner': '<resource_owner>', 'service_name': '<service-name>' } Data Lake Ad Hoc Reporting Databricks Job
- 21.
- 22. Delta Time Travel MergeAuto-optimize
- 23. Databricks for Analyticsand Data Science
- 24. Analytics Use Cases Exploratoryand root cause analysis Analysis for Strategic Decisions POC for new metrics and business logic Creating and refreshing ad-hoc datasets Team Onboarding Templates
- 25. Big Wins: Analytics Self-serviceCollaboration
- 26. Data Science UseCases Exploration, Sizing Feature generation Model training Scoring Experiments Analyzing results Model serving
- 27. Big Wins: DataScience Faster local stack to cloud cycle No infrastructure overhead Increased ML adoption across teams Governance & Tracking
- 28.
- 29. Key Takeaways Delivery timereduced by 30% Decreased infrastructure costs by 60% Databricks used by 50% of all Atlassians Reduced Data team dependencies by more than 70%
- 30.
- 31. Feedback Your feedback isimportant to us Don’t forget to rate and review the sessions