Downloaded 21 times
Uploaded byKris Buytaert
Open Source Monitoring in 2015
Kris Buytaert discusses the evolution of monitoring tools from #MonitoringSucks to #MonitoringLove. He describes how early monitoring tools like Nagios and Zenoss were difficult to configure and maintain. Newer tools like Icinga, Sensu, Graphite, Grafana, and Riemann provide modular components that are easier to automate and integrate. Logs can now be monitored alongside metrics using tools like Graylog and ELK. Packetbeat allows monitoring application performance. The monitoring landscape continues improving with better automation, integration, and reduced alert fatigue.
Open Source Monitoring in 2015
- 1.
- 2. Kris BuytaertKris Buytaert ●I used to be a Dev,I used to be a Dev, ● Then Became an OpThen Became an Op ● Chief Trolling Officer and Open SourceChief Trolling Officer and Open Source Consultant @inuits.euConsultant @inuits.eu ● Everything is an effing DNS ProblemEverything is an effing DNS Problem ● Building Clouds since before the bookstoreBuilding Clouds since before the bookstore ● Organising ConferencesOrganising Conferences ● Evangelizing devopsEvangelizing devops
- 3. An opinionated talkabout the Open SourceAn opinionated talk about the Open Source Monitoring tooling landscapeMonitoring tooling landscape In which I hope to learn from YOUIn which I hope to learn from YOU
- 4. #devops=~C(L)AMS#devops=~C(L)AMS ● CultureCulture ● (Lean)(Lean) ●AutomationAutomation ● Monitoring and MeasurementMonitoring and Measurement ● SharingSharing Damon Edwards and John WillisDamon Edwards and John Willis Gene KimGene Kim
- 5. Monitoring is usuallyanMonitoring is usually an aftertoughtaftertought ENOBUDGET, ENOTIMEENOBUDGET, ENOTIME
- 6. An 2008 OLSPaperAn 2008 OLS Paper ● We have bloated Java toolsWe have bloated Java tools ● Some open Core stufSome open Core stuf ● DYI folks want traditional NagiosDYI folks want traditional Nagios ● DBA RequiredDBA Required
- 7. #monitoringsucks#monitoringsucks ● John Vincent(@lusis), june 2011John Vincent (@lusis), june 2011 ● A sub #devops movementA sub #devops movement ● https://github.com/monitoringsucks/https://github.com/monitoringsucks/
- 8. Why #monitoringsucksWhy #monitoringsucks ●Manual config (gui)Manual config (gui) ● Not in sync with realityNot in sync with reality ● Hosts onlyHosts only ● Services sometimesServices sometimes ● Aplication neverAplication never ● Chaos or out of sync with realityChaos or out of sync with reality ● Alert FatigueAlert Fatigue
- 9. Let's forget aboutLet'sforget about ● Tools with no (stable) APITools with no (stable) API ● Tools with strong focus on GUITools with strong focus on GUI ● Unless you are an SME with < 100 nodesUnless you are an SME with < 100 nodes ● Zenoss, Hyperic, GroundWork, ....Zenoss, Hyperic, GroundWork, .... ● P.S. : don't even mention proprietary software to meP.S. : don't even mention proprietary software to me
- 10. What we wantWhatwe want ● Small , well suited componentsSmall , well suited components • CollectCollect • Transport / MangleTransport / Mangle • StoreStore • AnalyseAnalyse • Act / AlertAct / Alert • VisualizeVisualize
- 11. #monitoringlove#monitoringlove • • Ulf Mansson #devopsdaysRome 2011Ulf Mansson #devopsdays Rome 2011 • A new era of toolingA new era of tooling • #monitoringlove hacksessions @inuits#monitoringlove hacksessions @inuits • #monitorama#monitorama
- 15. IcingaIcinga • 2009 Fork2009 Fork • Iconsider Nagios deadI consider Nagios dead • Vibrant Community (or they stalk me)Vibrant Community (or they stalk me) • Throw great parties in NurnbergThrow great parties in Nurnberg • Nobody can pronounce it anyhowNobody can pronounce it anyhow • https://github.com/Inuits/puppet-icinga/https://github.com/Inuits/puppet-icinga/
- 16.
- 17. #monitoringlove#monitoringlove But the lovewas about :But the love was about :
- 18. SensuSensu ● Awesome fornon staticAwesome for non static environmentsenvironments ● Scaling a clustered RabbitMQ ?Scaling a clustered RabbitMQ ? ● This is Europe, U no do cloudThis is Europe, U no do cloud
- 19. Automation ofAutomation of #monitoring#monitoring broughtbackbrought back thethe #love#love
- 20. Monitoring aMonitoring aserviceservice vsvs Monitoring aMonitoring a ServiceService
- 21. definition of done:definitionof done: monitored and in productionmonitored and in production
- 22. A software projectis not doneA software project is not done untill your last end user is deaduntill your last end user is dead
- 23. Culture,Culture, Automation,Automation, Measurement :Measurement : measureall the thingsmeasure all the things SharingSharing
- 24. Deploy StatisticsDeploy Statistics ●Time To DeployTime To Deploy ● DeployDeploy FrequencyFrequency ● LifecycleLifecycle frequencyfrequency ● Map to otherMap to other metricsmetrics
- 25. CollectD all themetrics,CollectD all the metrics, at high intervalsat high intervals
- 26.
- 27. Self ServiceSelf Service Gdashbased pipelinesGdash based pipelines Puppetized Templates (wip)Puppetized Templates (wip)
- 28.
- 29.
- 30. Graphite++Graphite++ ● DashboardsDashboards • GrafanaGrafana ● Engines:Engines : • InfluxDBInfluxDB • CyaniteCyanite
- 31. Triggers on GraphsTriggerson Graphs ● Export Java MetricsExport Java Metrics ● JMXTransJMXTrans ● Export JMXConfigsExport JMXConfigs ● Configure NRPE CheckConfigure NRPE Check ● Export NagiosCheckExport NagiosCheck ● Collect JMX Exports onCollect JMX Exports on JMXTransNodeJMXTransNode ● Graph EmGraph Em Collect Icinga ConfigsCollect Icinga Configs on Icingaon Icinga
- 32. AggregationAggregation ● Alert onstreamsAlert on streams ● Alert on aggregated metricsAlert on aggregated metrics
- 35. RiemannRiemann ● I stilldon't get it ?I still don't get it ? ● Distributed TopDistributed Top ● Do you like Clojure ?Do you like Clojure ? ● Riemann Health plugin ?Riemann Health plugin ? ● s/riemann-health/collectd/g;s/riemann-health/collectd/g; ● Output to graphiteOutput to graphite
- 36. Graphs to KnowledgeGraphsto Knowledge SkylineSkyline • OculusOculus • Creating Information out of this dataCreating Information out of this data • Big dataBig data • Machine LearningMachine Learning
- 37. But I havelog files..But I have log files..
- 41. Logs and MetricsLogsand Metrics ● Graylog2Graylog2 ● ELSA (Enterprise Log Search andELSA (Enterprise Log Search and Archive)Archive) ● ELK StackELK Stack
- 42. ● Collect fromCollectfrom anywhereanywhere ● FilterFilter ● Send anywhereSend anywhere
- 44. APMAPM But what aboutmy apps ?But what about my apps ? Half the world cheers about SAASHalf the world cheers about SAAS tools :(tools :(
- 45. PacketbeatPacketbeat ● Traffic FlowTrafficFlow through networkthrough network ● TransactionsTransactions causing errroscausing errros ● SQL per HTTPSQL per HTTP ● API call usageAPI call usage
- 46.
- 47. So your DCfailsSo your DC fails Whom to alert when ?Whom to alert when ?
- 49. 'New' kids onthe block'New' kids on the block ● FlapjackFlapjack flapjack.ioflapjack.io monitoring notification routing +monitoring notification routing + event processing systemevent processing system ● OpenDutyOpenDuty github.com/szechuen/OpenDutygithub.com/szechuen/OpenDuty Duty managementDuty management
- 50. My Alerting StrategyMyAlerting Strategy Is still in betaIs still in beta
- 51. And back :(Andback :( In 2014 I`m still running the same check forIn 2014 I`m still running the same check for - service registration (consul)- service registration (consul) - high availability (pacemaker/corosync)- high availability (pacemaker/corosync) - monitoring (icinga)- monitoring (icinga)
- 52. But I lovewhere Monitoring is headingBut I love where Monitoring is heading We have much less false positivesWe have much less false positives And we have a Maintainable Monitoring InfraAnd we have a Maintainable Monitoring Infra KindaKinda
- 53. ContactContact Kris.Buytaert@inuits.euKris.Buytaert@inuits.eu Further ReadingFurther Reading @krisbuytaert@krisbuytaert http://www.krisbuytaert.be/blog/http://www.krisbuytaert.be/blog/ http://www.inuits.eu/http://www.inuits.eu/ InuitsInuits Duboistraat50Duboistraat 50 2060 Antwerpen2060 Antwerpen BelgiumBelgium 891.514.231891.514.231 +32 475 961221+32 475 961221