Download as PDF, PPTX
Elasticsearch Data Analyses
- 1. Elasticsearch Elasticsearch Timed Data Analyses ByAlaa Elhadba @aelhadba
- 2. Table of Contents -Hot-Cold Architecture - Data High Availability - Data design at large scale - Search Execution - Time framed indices - Aggregations
- 3.
- 4. Hot-Cold Architecture Hot DataNodes Perform indexing Hold most recent data Use SSD storage, Writing is an Intensive IO operation Cold Data Nodes Handle read only operations Can use large spinning disks
- 5. Hot-Cold Configuration node.box_type: hot elasticsearch.yaml Shard2 Node Shard 1 Node node.box_type: cold elasticsearch.yaml
- 6. Data Availability Availability Zone1 Availability Zone 2
- 7. Data Availability Availability Zone1 Availability Zone 2
- 8. Data Availability Availability Zone1 Availability Zone 2Availability Zone / Rack failure ? Shard Allocation Awareness
- 9. Shard Allocation Awareness AvailabilityZone 1 Availability Zone 2
- 10. Shard Allocation Awareness AvailabilityZone 1 Availability Zone 2 1 2 1 2 1 2 3 1 2 3
- 11. Shard Allocation Awareness cluster.routing.allocation.awareness.attributes:rack_1 ● Data replication is spanned across AZs ● No two copies of same shard on the same rack ● Elasticsearch is fully aware of shard distribution ● Awareness can be set based cluster or index ● Elasticsearch will prefer using local shards ● Always balance your nodes across AZs ● Routing Allocation Awareness can be updated on a live cluster cluster.routing.allocation.awareness.attributes: rack_2 Availability Zone 1 Availability Zone 2
- 12. Shard Allocation Awareness cluster.routing.allocation.awareness.attributes:rack_1 ● Data replication is spanned across AZs ● No two copies of same shard on the same rack ● Elasticsearch is fully aware of shard distribution ● Awareness can be set based cluster or index ● Elasticsearch will prefer using local shards ● Always balance your nodes across AZs ● Routing Allocation Awareness can be updated on a live cluster cluster.routing.allocation.awareness.attributes: rack_2 Availability Zone 1 Availability Zone 2
- 13. Shard Allocation Awareness cluster.routing.allocation.awareness.attributes:rack_1 ● Data replication is spanned across AZs ● No two copies of same shard on the same rack ● Elasticsearch is fully aware of shard distribution ● Awareness can be set based cluster or index ● Elasticsearch will prefer using local shards ● Always balance your nodes across AZs ● Routing Allocation Awareness can be updated on a live cluster cluster.routing.allocation.awareness.attributes: rack_2 Availability Zone 1 Availability Zone 2
- 14. Shard Allocation Awareness cluster.routing.allocation.awareness.attributes:rack_1 ● Data replication is spanned across AZs ● No two copies of same shard on the same rack ● Elasticsearch is fully aware of shard distribution ● Awareness can be set based cluster or index ● Elasticsearch will prefer using local shards ● Always balance your nodes across AZs ● Routing Allocation Awareness can be updated on a live cluster ● Use Forced Awareness to avoid the extra load of reallocation of missing shards cluster.routing.allocation.awareness.attributes: rack_2 Availability Zone 1 Availability Zone 2
- 15. Shard Allocation Awareness cluster.routing.allocation.awareness.attributes:rack_1 ● Data replication is spanned across AZs ● No two copies of same shard on the same rack ● Elasticsearch is fully aware of shard distribution ● Awareness can be set based cluster or index ● Elasticsearch will prefer using local shards ● Always balance your nodes across AZs ● Routing Allocation Awareness can be updated on a live cluster ● Use Forced Awareness to avoid the extra load of reallocation of missing shards cluster.routing.allocation.awareness.attributes: rack_2 Availability Zone 1 Availability Zone 2 Make sure you can handle the load with less nodes!
- 16. Forced Awareness ● Forcedawareness solves this problem by NEVER allowing copies of the same shard to be allocated to the same zone. ● Avoid extra of reallocating unassigned shards after rack failure. ● Allow no single point of failure for your system. ● Make sure you can handle the load with less nodes. cluster.routing.allocation.awareness.force.zone.values: zone1,zone2 cluster.routing.allocation.awareness.attributes: rack1,zone1
- 17. Data design atlarge scale
- 18. Searching Shard 4 Shard 2 Query Result Node Node Shard3 Node Shard 1 Node
- 19. Searching Shard 4 Shard 2 Query Result Node Node Shard3 Node Shard 1 Node How to avoid asking all shards ?
- 20. Searching Shard 4 Shard 2 Query Result Node Node Shard3 Node Shard 1 Node How to avoid asking all shards ? Routing I know my shards!
- 21. Routing PUT my_index/my_type/my_id?routing=shard1 GET my_index/_search?routing=shard1,shard2 ●Avoid calling all shards ● Dedicated shards per purpose ● Talk to one dedicated shard ● Eliminate Network Traffic ● Better Performance ● Handle sharding on your own
- 22. Routing PUT my_index/my_type/my_id?routing=shard1 GET my_index/_search?routing=shard1,shard2 ●Avoid calling all shards ● Dedicated shards per purpose ● Talk to one dedicated shard ● Eliminate Network Traffic ● Better Performance ● Handle sharding on your own But, Once in, Never out ● Routing must be always specified
- 23. Routing 1 2 31 2 3 1 2 21.06.2016 20.06.2016 19.06.2016
- 24. Routing 1 2 31 2 3 1 2 21.06.2016 20.06.2016 19.06.2016 I MUST KNOW EVERYTHING!
- 25.
- 26. Aliasing 1 2 31 2 3 1 2 21.06.2016 20.06.2016 19.06.2016 today yesterday 3_days_ago
- 27. Aliasing 1 2 31 2 3 1 2 21.06.2016 20.06.2016 19.06.2016 today yesterday 3_days_ago 1 2 3 22.06.2016
- 28. Aliasing 1 2 31 2 3 21.06.2016 20.06.2016 today yesterday 3_days_ago 1 2 3 22.06.2016
- 29. Aliasing 1 2 31 2 3 21.06.2016 20.06.2016 today yesterday 3_days_ago 1 2 3 22.06.2016 I MUST KNOW! it’s Better Performance
- 30. Aliasing 1 2 31 2 3 21.06.2016 20.06.2016 1 2 3 22.06.2016 It’s a Data Problem! today yesterday 3_days_ago
- 31. Aliasing + Routing 12 3 1 2 3 21.06.2016 20.06.2016 1 2 3 22.06.2016 It’s a Data Problem! today yesterday 3_days_agotoday_returns recent_returns
- 32. Aliasing + Routing+ Search IndexIndex Shard Alias Shard slice
- 33. Search Execution Preference Elasticsearchtargets shards and replicas in round-robin manner. Each shard is queried similarly _primary Query only primary shards (latest info from index or optimize for writing path) _primary_first Query primary first in available _replica Query replica shard only _replica_first Query replica first in available _local Query shards available on the current node _only_node:node_id Query a specific node _only_nodes:* Query only a set of nodes _prefer_node:node_id Query a prefered noe _shards:1,3 e,g _shards:1,3;_local Query specific shards with a preference PUT _search?preference=_replica
- 34.
- 35. Data Flow HOT ColdClosed Backed_up Trashed Time
- 36. Closing/Opening Index ➔ Closingan index ◆ Removes all shard allocations from the cluster ◆ But keeps the index data around ◆ Helps reduce the resources used on the cluster ◆ Consumes only disk space ➔ Opening an index ◆ Allows to open a closed index ◆ Note, those are not “milliseconds” time operation, opening an index can take a few seconds to a couple of minutes ◆ Flushing before closing will reduce the opening time
- 37. Index Templates - Orderallows you to override other templates - Settings allows you to scale anytime - Aliases can be defined on index creation
- 38.
- 39. Time framed indiceslifecycle 1. Use Index templates to generate mappings for new indices 2. Use aliases to decouple your application from data logic 3. Use hot nodes for fresh data 4. Move old data to cold nodes 5. Close old indices before deletion 6. Change your time frame at any point to scale (Monthly, Weekly….) 7. Use Routing if you have too many shards in a big cluster
- 40. Data Flow HOT ColdClosed Backed_up Trashed Time
- 41.
- 42.
- 43.
- 44.
- 45. Aggregation Query Better caching Fetchrelevant documents First segmentation Nested segmentation
- 46. Doc Values - Whydo we need this? - Sorting, Aggregations, Some Scripting - Doc Values - Build columnar style data structure on disk - Created at indexing time, stored as part of the segment - Read like other pieces of the Lucene index - Don't take up heap space - Uses file system cache - Default for not_analyzed string and numeric fields in 2.0+
- 47. Raw Fields - Usecustomer_name.raw for aggregations - Use customer_name for search
- 48.
- 49. Metrics Aggregations - AvgAggregation - Cardinality Aggregation - Extended Stats Aggregation - Max Aggregation - Min Aggregation - Percentiles Aggregation - Percentile Ranks Aggregation - Scripted Metric Aggregation - Stats Aggregation - Sum Aggregation - Top hits Aggregation - Value Count Aggregation
- 50.
- 51. Aggregation Search Shard 4 Shard2 Query Result Node Node Shard 3 Node Shard 1 Node
- 52. Scripted Metric Aggregation -Init_script Executed first. Allows initialization of variables. - map_script Executed once after each document is collected. - combine_script Executed once on each shard after document collection is complete. - reduce_script Executed once on the coordinating node after all shards have returned their results.
- 53. Buckets Aggregations - ChildrenAggregation - Date Histogram Aggregation - Date Range Aggregation - Filter Aggregation - Filters Aggregation - Global Aggregation - Histogram Aggregation - Missing Aggregation - Range Aggregation - Reverse nested Aggregation - Sampler Aggregation - Significant Terms Aggregation - Terms Aggregation
- 54.
- 55. Date Range Aggregation Don’tforget! Round your dates
- 56.
- 57.
- 58.
- 59.
- 60. Pipeline Aggregations Parent - Ableto compute new buckets or new aggregations to a parent aggregation. Sibling - Able to compute new buckets or new aggregation on the same level.
- 61. Siblings Aggregation - min_bucket -max_bucket - sum_bucket - avg_bucket - stats_bucket - extended_stats_bucket - percentiles_bucket
- 62.
- 63. Parent Pipeline Aggregation -moving_avg - derivative - cumulative_sum - bucket_script - bucket_selector - serial_diff
- 64.
- 65.
- 66.
- 67.
- 68.
- 69.
- 70.
- 71.