Download as ODP, PPTX
Uploaded byKris Buytaert
The influence of "Distributed platforms" on #devops
The document discusses the evolution of devops practices from the early 2000s to present day. It describes how early tools like openMosix helped distribute processes across nodes but had limitations that prevented widespread adoption. Linux-HA helped make high availability services more common by defining resources and constraints. This highlighted the need for applications to be adaptable and share state in a distributed manner. Private clouds initially failed to adopt configuration management, monitoring, and other devops practices. Today, containers are more widely used but often resemble virtual machines with multiple services and no standard way to connect or monitor them. Adopting microservices and devops fully requires changes across software, infrastructure, mindsets and organizations.
The influence of "Distributed platforms" on #devops
- 1. The influence of“distributed platforms” on devops Kris Buytaert @krisbuytaert
- 2. @KrisBuytaert@KrisBuytaert ● I usedto be a Dev,I used to be a Dev, ● Then Became an OpThen Became an Op ● CTO and Open Source Consultant @CTO and Open Source Consultant @inuits.euinuits.eu ● Everything is a DNS ProblemEverything is a DNS Problem ● Evangelizing devopsEvangelizing devops ● Organiser of #devopsdays, #cfgmgmtcamp, #loadays,Organiser of #devopsdays, #cfgmgmtcamp, #loadays, ….….
- 3. What's thisWhat's this devopsthingdevops thing anyhow ?anyhow ?
- 4. World , 200X-2009World, 200X-2009 Patrick Debois, Gildas Le Nadan, Andrew Clay Shafer, Kris Buytaert, JezzPatrick Debois, Gildas Le Nadan, Andrew Clay Shafer, Kris Buytaert, Jezz Humble, Lindsay Holmwood, John Willis, Chris Read, Julian Simpson, and lotsHumble, Lindsay Holmwood, John Willis, Chris Read, Julian Simpson, and lots of others ..of others .. Gent , October 2009Gent , October 2009 Mountain View , June 2010Mountain View , June 2010 Hamburg, Goteborg, Rome, ...Hamburg, Goteborg, Rome, ... 5th aniversary in Gent 20145th aniversary in Gent 2014 25+ events in 201625+ events in 2016
- 5. C(L)AMSC(L)AMS ● CultureCulture ● (Lean)(Lean) ●AutomationAutomation ● MeasurementMeasurement ● SharingSharing Damon Edwards and John WillisDamon Edwards and John Willis Gene KimGene Kim
- 6. The Old DaysTheOld Days ● ““Put this Code Live, here's a tarball” NOW!Put this Code Live, here's a tarball” NOW! ● What dependencies ?What dependencies ? ● No machines available ?No machines available ? ● What database ?What database ? ● Security ?Security ? ● High Availability ?High Availability ? ● Scalability ?Scalability ? ● My computer can't install this ?My computer can't install this ?
- 7. 10 days intooperations10 days into operations
- 8. 11 days intooperations11 days into operations
- 9.
- 10. openMosixopenMosix (+/- 2001-2005)(+/-2001-2005) ● 2.4 Kernel patch,2.4 Kernel patch, ● Loadbalance multiple similar processesLoadbalance multiple similar processes ● Both short (compilations) and longlived (calculations)Both short (compilations) and longlived (calculations) processesprocesses ● Parts of processes migrated to other nodesParts of processes migrated to other nodes ● OMFS allowed identical access to data from all nodesOMFS allowed identical access to data from all nodes
- 11. openMosix LimitationsopenMosix Limitations ●shared memoryshared memory • Limited type of applications could migrateLimited type of applications could migrate • Patches available but not successfulPatches available but not successful ● Filesystem accessFilesystem access oMFS : unstable => removedoMFS : unstable => removed
- 12. Lessons from openMosixLessonsfrom openMosix ● Applications need to be adaptedApplications need to be adapted ● Files need to be accessible (oMFS)Files need to be accessible (oMFS) ● Huge gap between developers and consumersHuge gap between developers and consumers => Limited working use cases=> Limited working use cases
- 13. Linux-HALinux-HA (2005- now)(2005-now) ● Hearbeat, Heartbeat v2, Pacemaker , CorosyncHearbeat, Heartbeat v2, Pacemaker , Corosync ● Define resource (OCF)Define resource (OCF) ● Define constraints, clones, colocation, …Define constraints, clones, colocation, … ● Long running services that should not stopLong running services that should not stop
- 14. Pacemaker ArchitecturePacemaker Architecture ●Stonithd : The Heartbeat fencing subsystem. ● Lrmd : Local Resource Management Daemon. Interacts directly with resource agents (scripts). ● pengine Policy Engine. Computes the next state of the cluster based on the current state and the configuration. ● cib Cluster Information Base. Contains definitions of all cluster options, nodes, resources, their relationships to one another and current status. Synchronizes updates to all cluster nodes. ● crmd Cluster Resource Management Daemon. Largely a message broker for the PEngine and LRM, it also elects a leader to co-ordinate the activities of the cluster. ● openais messaging and membership layer. ● heartbeat messaging layer, an alternative to OpenAIS. ● ccm Short for Consensus Cluster Membership. The Heartbeat membership layer.
- 15. Working with developersWorkingwith developers ● State vs StatelessState vs Stateless • Most applications have state somewhereMost applications have state somewhere • Discuss how to share/access stateDiscuss how to share/access state ● Data accessData access • Local filesystem is kingLocal filesystem is king • We needed distributed / shared filesystemsWe needed distributed / shared filesystems ● HealthHealth ● MetricsMetrics
- 16. Isolating ServicesIsolating Services ●1 application per VM1 application per VM • Easy configurationEasy configuration • No accidental changesNo accidental changes Anno 2005 – 2010 people called us crazyAnno 2005 – 2010 people called us crazy ● Multiple VM's with the same functions (A,B,001,002,256)Multiple VM's with the same functions (A,B,001,002,256) ● Isolate code from configIsolate code from config
- 17. Lessons from Linux-HALessonsfrom Linux-HA ● Applications need to be adaptedApplications need to be adapted ● Files need to be accessible (shared/distributed FS)Files need to be accessible (shared/distributed FS) ● Monitoring strategy needs changeMonitoring strategy needs change ● Good collaboration between developers and ops folksGood collaboration between developers and ops folks => Many general purpose use cases=> Many general purpose use cases ● Works better with an Open Source MindsetWorks better with an Open Source Mindset
- 18. Impact on Monitoring/MetricsImpacton Monitoring/Metrics ● HA , monitoring a serviceHA , monitoring a service • Monitoring the individual serviceMonitoring the individual service • Monitoring/Ensuring the global serviceMonitoring/Ensuring the global service ● Health status of an applicationHealth status of an application ● Metric collectionMetric collection ● Alerting based on metrics rather than only status checksAlerting based on metrics rather than only status checks
- 19. Back to 2016Backto 2016
- 20. Cloud Adoption is(s)lowCloud Adoption is (s)low ● Enterprises are afraid cloudEnterprises are afraid cloud • Security, Cost, Control, Stability, (insert more fud here)Security, Cost, Control, Stability, (insert more fud here) ● ““Private cloud” will solve thisPrivate cloud” will solve this ● Please fill in 4 word documents for for each VM you wantPlease fill in 4 word documents for for each VM you want
- 21. Failed (Private) CloudProjectsFailed (Private) Cloud Projects ● Identical copies of Bare Metal or vm's moved to the cloudIdentical copies of Bare Metal or vm's moved to the cloud ● No config managementNo config management ● No monitoringNo monitoring ● No resilienceNo resilience ● Legacy procedures (still 5 week wait for a VM)Legacy procedures (still 5 week wait for a VM) ● No API usageNo API usage
- 22. These DaysThese Days ●““Put this Code Live, here's Docker Container” NOW!Put this Code Live, here's Docker Container” NOW! ● No machines available ?No machines available ? ● What database ?What database ? ● Security ? What distro is this even ?Security ? What distro is this even ? ● High Availability ?High Availability ? ● Scalability ?Scalability ? ● How do we monitor his ?How do we monitor his ? ● How did you build this ?How did you build this ?
- 24. A typical EnterpriseContainerA typical Enterprise Container ● No different from a full vmNo different from a full vm ● Multiple services running inMultiple services running in on containeron container ● Ssh is the default connectionSsh is the default connection
- 25.
- 26. IT Departments havenot adapted,IT Departments have not adapted, 'Shadow'-IT is winning'Shadow'-IT is winning
- 27. Elephant , MonolithElephant, Monolith vsvs MicroservicesMicroservices
- 28. Can we Slicethe Elephants ?Can we Slice the Elephants ? {Devops/Cloud/Container}{Devops/Cloud/Container} adoption requires changeadoption requires change • In software architectureIn software architecture • In codeIn code • In infrastructureIn infrastructure • In mindsetIn mindset • In organisationIn organisation • In managementIn management
- 29. ContactContact Kris Buytaert Kris.Buytaert@inuits.euKrisBuytaert Kris.Buytaert@inuits.eu Further ReadingFurther Reading @krisbuytaert@krisbuytaert http://www.krisbuytaert.be/blog/http://www.krisbuytaert.be/blog/ http://www.inuits.eu/http://www.inuits.eu/ Inuits.euInuits.eu Essensteenweg 31Essensteenweg 31 BrasschaatBrasschaat BelgiumBelgium 891.514.231891.514.231 +32 475 961221+32 475 961221