Download to read offline
Uploaded byGetInData
Kubernetes and real-time analytics - how to connect these two worlds with Apache Flink? - Albert Lewandowski, GetInData
The document discusses the integration of Kubernetes and real-time analytics through Apache Flink, highlighting concepts such as Kubernetes operators, custom resource definitions, and Flink's unique savepoints. It includes use cases, the deployment process on Kubernetes, and emphasizes the importance of observability through tools like Prometheus. Additionally, it provides practical insights for setting up a local environment and monitoring performance, along with opportunities for collaboration in the field.
Kubernetes and real-time analytics - how to connect these two worlds with Apache Flink? - Albert Lewandowski, GetInData
- 1. Kubernetes and real-time analytics Howto connect these two worlds with Apache Flink? Author: Albert Lewandowski
- 2. © Copyright. Allrights reserved. Not to be reproduced without prior written consent. About me ● Big Data DevOps Engineer - GetInData ● Focused on infrastructure, cloud, Big Data, AI, scalable web applications ● Certified Google Cloud Architect ● Certified Kubernetes Administrator
- 3. © Copyright. Allrights reserved. Not to be reproduced without prior written consent. Content ● Principles in the Big Data world on Kubernetes ● Why real-time data streaming? ● Different faces of Apache Flink. ● Flink and Kubernetes - real life scenarios. ● Observability of the platform. ● Quick start on your computer.
- 4.
- 5. © Copyright. Allrights reserved. Not to be reproduced without prior written consent. What is Kubernetes? Open-source platform for managing containerized workloads and services
- 6. © Copyright. Allrights reserved. Not to be reproduced without prior written consent. Kubernetes - Operators Method of deploying and managing app Automated provisioning of resources One setup for multiple environments Examples: pulsar-operator, postgres-operator, prometheus-operator
- 7. © Copyright. Allrights reserved. Not to be reproduced without prior written consent. Kubernetes - Custom Resource Definitions Defining custom APIs as add-ons Dynamic registration with Kubernetes API CRDs can be accessed with kubectl A CRD represents the desired state and an operator makes it happen.
- 8. © Copyright. Allrights reserved. Not to be reproduced without prior written consent. What is Apache Flink? Flink is an open-source stream processing framework that supports both batch processing and data streaming programs Flink’s Savepoints are different from Checkpoints in a similar way that backups are different from recovery logs in traditional database systems. A savepoint is a consistent image of the execution state of a streaming job State of the Flink job
- 9. © Copyright. Allrights reserved. Not to be reproduced without prior written consent. What is Apache Flink? Job Diagram State of Flink job Diagram
- 10. © Copyright. Allrights reserved. Not to be reproduced without prior written consent. Apache Flink Roadmap Source: Roadmap - Apache Flink
- 11. © Copyright. Allrights reserved. Not to be reproduced without prior written consent. Perception Business logic CI/CD Idempotency Reprocessing Explainability Monitoring Testing Serving Infrastructure Data Ingestion Security
- 12. © Copyright. Allrights reserved. Not to be reproduced without prior written consent. Reality Business logic CI/CD Idempotency Reprocessing Explainability Monitoring Testing Serving Infrastructure Data Ingestion Security
- 13.
- 14. © Copyright. Allrights reserved. Not to be reproduced without prior written consent. Data Streaming vs. Batch Events 1 2 3 4 5 6 Batch Stream
- 15. © Copyright. Allrights reserved. Not to be reproduced without prior written consent. Use cases User activity Fraud detection Logistics Industrial IoT Location data Recommendations
- 16. © Copyright. Allrights reserved. Not to be reproduced without prior written consent. Use case - Kcell - Telecom company in Kazakhstan 500K Events / second 10M Subscribers 165K Events / s 10M Subscribers 22.5 TB / month 2020 2018 40 TB / month
- 17. © Copyright. Allrights reserved. Not to be reproduced without prior written consent. Use case - Kcell SMS events Voice usage events Data usage events Roaming events Location events Input Process Actions
- 18. © Copyright. Allrights reserved. Not to be reproduced without prior written consent. Use case - Kcell - some scenarios for Flink Balance Top Up Case If subscriber top-ups her balance too often in short period of time. We can offer her a less expensive tariff or auto-payment services. Fraud case in roaming Send an email to the anti-fraud unit if subscriber registered in roaming but his balance at the moment is equal to 0. This situation is impossible in standard case. Automatic SIM card activation Send an email to the anti-fraud unit if subscriber registered in roaming but his balance at the moment is equal to 0. This situation is impossible in standard case. Dealer Motivation Case Trigger bonus for a dealer when we discover that purchase happened attributable to him/her.
- 19. Apache Flink One tool,multiple versions
- 20. © Copyright. Allrights reserved. Not to be reproduced without prior written consent. One tool, multiple languages Java 8 or 11 Python SQL Scala 2.11 or 2.12
- 21. © Copyright. Allrights reserved. Not to be reproduced without prior written consent. Where should I install? Standalone Kubernetes YARN cluster ● CICD process ● Service Discovery - monitoring with Prometheus ● Scalability ● Managing resources ● A/B Testing
- 22. © Copyright. Allrights reserved. Not to be reproduced without prior written consent. High Availability of Flink JobManager level Storage level ● ZooKeeper ● Kubernetes (beta) ● High Availability of storage to/from which Flink writes/reads savepoints and checkpoints ● Performance of storage Job Strategy ● Data reprocessing policy ● How to deploy new job?
- 23. © Copyright. Allrights reserved. Not to be reproduced without prior written consent. Flink K8S Operator Kubernetes Operator for Apache Flink Ververica Platform Native Kubernetes - Apache Flink CRDs Yes Yes No No CICD Kubernetes API Kubernetes API REST API or Web UI Kubernetes API Installation Helm chart or raw Kubernetes manifests Helm chart or raw Kubernetes manifests Helm chart or raw Kubernetes manifests No need to install any component SQL Editor No No Yes No Dependencies No No Persistence volume for database Object storage for artifactory No Status beta beta production beta
- 24. Flink + Kubernetes= ? Overview in the article here
- 25. © Copyright. Allrights reserved. Not to be reproduced without prior written consent. Why Flink on Kubernetes? Simpler deployment process Flexible jobs management Simple Service Discovery - Prometheus Flexible testing
- 26. © Copyright. Allrights reserved. Not to be reproduced without prior written consent. Installation & Configuration Helm A package manager for Kubernetes CICD tool Example: Gitlab CI Kubernetes API Flink jobs
- 27. © Copyright. Allrights reserved. Not to be reproduced without prior written consent. Testing Incubating Mode A/B Testing Blue Green Deployment Production data Production job Job Incubating mode Separated output Standard output Dedicated TaskManagers Dedicated TaskManagers savepoi nt Proxy Flink Job #1 Flink Job #2 Result #1 Result #2
- 28. © Copyright. Allrights reserved. Not to be reproduced without prior written consent. Deployment process Versioning images Unit & Integration tests Git Flow Deployment process Monitoring
- 29. © Copyright. Allrights reserved. Not to be reproduced without prior written consent. Kubernetes aspects Dedicated namespaces Secured access to Flink (RBAC) Configuration files Storage for savepoints&checkpoints Resources Secrets Network performance
- 30. © Copyright. Allrights reserved. Not to be reproduced without prior written consent. Self-healing and autoscaling Scale based on metrics Flink restarts External tool for fixing Re-create cluster Automate manual tasks
- 31. © Copyright. Allrights reserved. Not to be reproduced without prior written consent. Job Cluster & Session Cluster Job Cluster Session Cluster Full set of Flink cluster for each individual job Standalone Flink cluster on Kubernetes Short running tasks Ad-hoc queries Long running tasks Separate images for different jobs
- 32. © Copyright. Allrights reserved. Not to be reproduced without prior written consent. Stories from production ● Automate in the beginning ● CICD pipeline is a must ● Verify JVM metrics ● Test different Flink configurations to get the best performance and no restarts ● Secure access to Flink jobs ● Get logs from Flink TMs and JMs
- 33.
- 34. © Copyright. Allrights reserved. Not to be reproduced without prior written consent. How to start locally? ● Minikube / Docker Desktop or any different local K8s env ● Ververica Platform ● Locally started Kafka cluster or use a Datagen APACHE FLINK KUBERNETES STREAMING SQL
- 35.
- 36. © Copyright. Allrights reserved. Not to be reproduced without prior written consent. Observability Observability is about measuring how well internal states of the system can be inferred from knowledge of its external outputs (according to the control theory).
- 37. © Copyright. Allrights reserved. Not to be reproduced without prior written consent. Part One: Metrics Get metrics from environment and application - but how?
- 38. © Copyright. Allrights reserved. Not to be reproduced without prior written consent. Prometheus - Kubernetes-native solution open-source systems monitoring and alerting toolkit joined the Cloud Native Computing Foundation in 2016 as the second hosted project, after Kubernetes a lot of exporters you can write your own easily mature ecosystem PushGateway, Blackbox, AlertManager, etc.
- 39. © Copyright. Allrights reserved. Not to be reproduced without prior written consent. Prometheus - simple or complex High Availability? Simple Complex Example solutions: Cortex (above), Thanos, M3DB
- 40. © Copyright. Allrights reserved. Not to be reproduced without prior written consent. Pull vs. push-based monitoring Pull Push Collector takes metrics Agents push metrics Workload on central poller increases with the number of devices polled. Polling task fully distributed among agents, resulting in linear scalability. Polling protocol can potentially open up system to remote access and denial of service attacks. Push agents are inherently secure against remote attacks since they do not listen for network connections. Flexible: poller can ask for any metric at any time. Relatively inflexible: pre-determined, fixed set of measurements are periodically exported.
- 41. © Copyright. Allrights reserved. Not to be reproduced without prior written consent. Prometheus - Stories service discovery simple on k8s limited security archived data how old data is required? monitor monitoring
- 42. © Copyright. Allrights reserved. Not to be reproduced without prior written consent. Part Two: Logs analytics 1. Get logs from app or environment. 2. Save logs. 3. Query them. 4. Make your system self-healing and discover what’s happening inside your platform.
- 43. © Copyright. Allrights reserved. Not to be reproduced without prior written consent. Logs analytics - which tool should I choose? Logs Analytics for Developers Logs Analytics for Business Loki ElasticSearch
- 44. © Copyright. Allrights reserved. Not to be reproduced without prior written consent. ELK vs. Loki ELK Loki + Promtail/Fluentd Indexing Keys and content of each key Only labels Query language Query DSL or Lucene QL LogQL Tool for data visualisation Kibana Grafana Query performances Faster due to indexed all the data Slower due to indexing only labels Resource requirements Higher due to the need of indexing Lower due to index only labels
- 45. © Copyright. Allrights reserved. Not to be reproduced without prior written consent. What about alerts? Alerts signify that a human needs to take action immediately in response to something that is either happening or about to happen, in order to improve the situation.
- 46.
- 47. © Copyright. Allrights reserved. Not to be reproduced without prior written consent. Flink - Complex Event Processing Article. Codebase for example.
- 48. © Copyright. Allrights reserved. Not to be reproduced without prior written consent. DevOps best practises Article.
- 49. © Copyright. Allrights reserved. Not to be reproduced without prior written consent. Kubernetes - first setup ● Minikube ● Kind ● Use Kubernetes service from public cloud provider like AWS, GCP, Azure during free tier
- 50. © Copyright. Allrights reserved. Not to be reproduced without prior written consent. Kubernetes + Flink - Operator $ kubectl create -f https://raw.githubusercontent.com/lyft/flinkk8soperator/v0.5.0/deploy/crd.yaml $ kubectl create -f https://raw.githubusercontent.com/lyft/flinkk8soperator/v0.5.0/deploy/namespace.yaml $ kubectl create -f https://raw.githubusercontent.com/lyft/flinkk8soperator/v0.5.0/deploy/role.yaml $ kubectl create -f https://raw.githubusercontent.com/lyft/flinkk8soperator/v0.5.0/deploy/role-binding.yaml $ kubectl create -f https://raw.githubusercontent.com/lyft/flinkk8soperator/v0.5.0/deploy/flinkk8soperator.yaml Verify if it works: $ kubectl -n flink-operator get po Run the example job: $ kubectl create -f https://raw.githubusercontent.com/lyft/flinkk8soperator/v0.5.0/examples/wordcount/flink-operator-custom-resou rce.yaml Verify if it is running and its status: $ kubectl get flinkapplication.flink.k8s.io -n flink-operator wordcount-operator-example -o yaml Requirements: Kubernetes cluster, kubectl
- 51. © Copyright. Allrights reserved. Not to be reproduced without prior written consent. Kubernetes + Ververica Platform Install Ververica Platform locally with Helm $ helm repo add ververica https://charts.ververica.com $ helm install vvp ververica/ververica-platform $ helm install vvp ververica/ververica-platform --set acceptCommunityEditionLicense=true Verify if Ververica is up $ kubectl get po Access the web user interface and REST API $ kubectl port-forward service/vvp-ververica-platform 8080:80 Do you want to test Flink SQL feature? Use Flink Faker (a data generator source connector) https://github.com/knaufk/flink-faker/ It requires changing used image for vvp-gateway. Requirements: Kubernetes cluster, kubectl, Helm
- 52. © Copyright. Allrights reserved. Not to be reproduced without prior written consent. Join Us! Data Engineer Spark, Kafka, Airflow, public cloud Link Backend Engineer Java / Scala, microservices Link MLOps Engineer MLOps tools, Python, public cloud Link DevOps / SRE GCP, Terraform, Prometheus Link
- 53. © Copyright. Allrights reserved. Not to be reproduced without prior written consent. Q&A
- 54. © Copyright. Allrights reserved. Not to be reproduced without prior written consent. Contact details albert.lewandowski@getindata.com LinkedIn: https://www.linkedin.com/in/albert-lewandowski
- 55. Thank you foryour attention!