Download as PDF, PPTX
Uploaded byDatabricks
Using Databricks as an Analysis Platform
The document discusses the use of Databricks as an analytics platform, highlighting features such as ETL workload management, various data types supported, and the integration of 3rd party data. It also emphasizes the importance of automation using tools like Airflow for workflow management, along with best practices for data handling and storage. Additionally, the document outlines YipitData's research offerings and encourages feedback and job applications.
Using Databricks as an Analysis Platform
- 2. Using databricks as ananalysis platform Anup Segu
- 3. Agenda Extending databricks toprovide a robust analytics platform Why a platform? What is in our platform?
- 4. Feedback Your feedback isimportant to us. Don’t forget to rate and review the sessions.
- 5.
- 6. YipitData Answers KeyInvestor Questions ▪ 70+ research products covering U.S. and international companies ▪ Email reports, excel files, and data downloads ▪ Transaction data, web data, app data, targeted interviews, and adding more ▪ Clients include over 200 investment funds and fortune 500 companies ▪ 53 data analysts and 3 data engineers ▪ 22 engineers total ▪ We are rapidly growing and hiring!
- 7. About Me ▪ SeniorSoftware Engineer ▪ Manage platform for ETL workloads ▪ Based out of NYC ▪ linkedin.com/in/anupsegu
- 8. We Want AnalystsTo Own The Product Data Collection Data Exploration ETL Workflows Report Generation www SELECT * FROM ...
- 10.
- 11. Python Library InsideNotebooks
- 12.
- 13. Wide range of tablesizes and schemas 1 PB Compressed Parquet 60 K Tables 1.7 K Databases
- 14. Readypipe: From URLsTo Parquet ▪ Repeatedly capture a snapshot of the website ▪ Websites frequently change ▪ Makes data available quickly for analysis
- 15. Glue Metastore Streaming AsJSON Is Great ▪ Append only data in S3 ▪ We don’t know the schema ahead of time ▪ Only flat column types s3://{json_bucket}/{project_name} /{table}/... JSON Bucket Parquet Bucket Kinesis Firehose
- 16. Parquet Makes Data“Queryable” ▪ Create or update databases, tables, and schemas as needed ▪ Partitioned by the date of ingestion ▪ Spark cluster subscribed to write events s3://{parquet_bucket}/{project_name} /{table}/dt={date}... JSON Bucket Kinesis Firehose Glue Metastore Parquet Bucket
- 17. Compaction = GreaterPerformance ▪ Insert files into new S3 locations ▪ Update partitions in Glue ▪ Pick appropriate column lengths for optimal file counts s3://{parquet_bucket}/{project_name} /{table}/compacted/dt={date}... JSON Bucket Kinesis Firehose Glue Metastore Parquet Bucket
- 18. With 3rd PartyData, We Strive for Uniformity Various File Formats Permissions Challenges Data Lineage Data Refreshes 403 Access Denied
- 19. Databricks Helps Manage3rd Party Data ▪ Upload files and convert to parquet with additional metadata ▪ Configure data access by assuming IAM roles within notebooks
- 20.
- 21. Table: Database +Name + Data
- 23. Table Hygiene PaysOff Validate table naming conventions Keep storage layer organized Maintain prior versions of tables Automate table maintenance
- 24. However, Our TeamIs Focused On Analysis so best practices are built into “create_table”
- 25.
- 26. Wide Range OfOptions For Spark Clusters Hardware Permissions Spark Configuration Driver instance Metastore Runtime Worker instances S3 access Spark properties EBS Volumes IAM Roles Environment Variables
- 27. Wide Range OfOptions For Spark Clusters Hardware Permissions Spark Configuration Driver instance Metastore Runtime Worker instances S3 access Spark properties EBS Volumes IAM Roles Environment Variables
- 28. T-Shirt Sizes ForClusters ▪ 3 r5.xlarge instances ▪ Warm instance pool for fast starts ▪ 10 r5.xlarge instances ▪ Larger EBS volumes available if needed “MEDIUM”“SMALL” ▪ 30 r5.xlarge instances Larger EBS volumes for heavy workloads “LARGE” Standard IAM Roles, Metastore, S3 access, and Environment Variables
- 29. Launch Spark JobsWith Ease
- 30. Names Map ToDatabricks Configurations
- 31. Databricks Does TheHeavy Lifting ▪ Provisions compute resources via a REST API ▪ Scales instances for cluster load ▪ Applies a wide range of spark optimizations
- 32.
- 33. Airflow Is OurPreferred ETL Tool
- 34. Airflow Is OurPreferred ETL Tool Requires someone to manage this code
- 35. We use thedatabricks API to construct DAGs programmatically +
- 36. 1 DAG =1 Folder, 1 Task = 1 Notebook
- 37. Templated Notebooks ForDAGs /folder - commands - notebook_a - notebook_b - notebook_c
- 38. Translate Notebooks IntoDAG files /api/2.0/workspace/list /api/2.0/workspace/export
- 39. Automatically Create Workflows ▪Pipelines are deployed without engineers ▪ Robust logging and error handling ▪ Easy to modify DAGs ▪ All happens within databricks Task A Task B Task C
- 40.
- 41.
- 42.
- 43. Visualize Logs InNotebooks
- 44. A Platform InvitesNew Solutions ▪ Establish standard queries and notebooks ▪ Trigger one DAG from one another ▪ Trigger reporting processes after ETL jobs
- 45. Thank You Interested inworking with us? We are hiring! yipitdata.com/careers
- 46. Feedback Your feedback isimportant to us. Don’t forget to rate and review the sessions.
- 47.
- 48.
- 49.
- 50.
- 51. Appendix II: TableCreation Code
- 52. Capturing metadata withsource data
- 53.
- 54. Creating a table(cont.)
- 55. Creating a table(cont.)
- 56. Creating a table(cont.)
- 57. Appendix III: DatabricksJobs Code
- 58.
- 59. Create a DatabricksJob (cont.)
- 60.
- 61.
- 62. Automatic DAG Creation(cont.)
- 63. Automatic DAG Creation(cont.)
- 64. Automatic DAG Creation(cont.)