Download as PDF, PPTX
Uploaded byJoe Stein
Get started with Developing Frameworks in Go on Apache Mesos
The document provides an overview of developing frameworks in Go on Apache Mesos, including its architecture, the role of frameworks, and the configuration of resources and constraints. It discusses the data center operating system (DCOS) built on Mesos, which allows for the deployment and management of applications across cloud resources. Key components like the scheduler, executor, and various APIs are highlighted for effective framework development.
Get started with Developing Frameworks in Go on Apache Mesos
- 1. Get started withDeveloping Frameworks in Go on Apache Mesos
- 2. Joe Stein CEO, Co-Founderof Elodina Inc, Software Platform focused on simplified operations of data oriented systems by providing the tools and support necessary to fully utilize their data center resources. LinkedIn: http://linkedin.com/in/charmalloc Twitter : @allthingshadoop
- 3. ◉ What isMesos? ◉ Framework = (Scheduler + Executor) ◉ Framework API Examples ◉ More Mesos! Overview
- 4.
- 5. Origins ◉ A CommonSubstrate for Cluster Computing https://people.csail.mit. edu/matei/papers/2009/hotcloud_nexus.pdf ◉ Mesos: A Platform for Fine-Grained Resource Sharing in the Data Center http://static.usenix.org/event/nsdi11/tech/full_papers/Hindman_new.pdf ◉ Google Borg - https://research.google.com/pubs/pub43438.html ◉ Google Omega: flexible, scalable schedulers for large compute clusters http: //eurosys2013.tudos.org/wp-content/uploads/2013/paper/Schwarzkopf.pdf
- 10.
- 11.
- 12. Data Center OperatingSystem Mesosphere’s Data Center Operating System (DCOS) is an operating system that spans all of the machines in a datacenter or cloud and treats them as a single computer, providing a highly elastic and highly scalable way of deploying applications, services, and big data infrastructure on shared resources. DCOS is based on Apache Mesos and includes a distributed systems kernel with enterprise-grade security. It also includes a set of core system services, such as a native Marathon instance to manage processes and installable services, and Mesos-DNS for service discovery. DCOS provides a web interface and a command-line interface (CLI) to manage the deployment and scale of applications.
- 16. Framework = (Scheduler+ Executor)
- 18.
- 19.
- 20. Marathon https://github.com/mesosphere/marathon Cluster-wide init andcontrol system for services in cgroups or docker based on Apache Mesos
- 21.
- 22. Use YAML toConfigure Your Deploy
- 23. Framework API &Examples
- 24. Mesos Protos https://github.com/apache/mesos/blob/master/include/mesos/mesos.proto Everything isgood to understand but here is a good place to start ◉ FrameworkInfo ◉ TaskInfo ◉ TaskState ◉ MasterInfo ◉ SlaveInfo
- 25. Mesos Framework DevelopmentGuide http://mesos.apache.org/documentation/latest/app-framework-development-guide/ Go Bindings https://github.com/mesos/mesos-go ◉ Scheduler API ○ registered, reregistered, disconnected ○ resourceOffers, offerRescinded, statusUpdate, frameworkMessage ○ slaveLost, executorLost, error ◉ Executor API ○ registered, reregistered, disconnected ○ launchTask, killTask, frameworkMessage
- 26. Go Kafka ClientMesos https://github.com/elodina/go-kafka-client-mesos
- 27. More About theScheduler
- 28.
- 29.
- 30.
- 31. Task Reconciliation http://mesos.apache.org/documentation/latest/reconciliation/ It isthe responsibility of Mesos (scheduler driver / Master) to ensure that the framework is notified when a disconnection, and subsequent (re-)registration occurs. At this point, the scheduler should perform task state reconciliation.
- 32.
- 33.
- 35. More Schedulers toCheck out ◉ https://github.com/elodina/statsd-mesos-kafka ◉ https://github.com/elodina/syslog-service ◉ https://github.com/mesos/kafka ◉ https://github.com/elodina/exhibitor-mesos-framework ◉ https://github.com/elodina/go-kafka-client-mesos ◉ https://github.com/elodina/datastax-enterprise-mesos ◉ https://github.com/elodina/zipkin-mesos-framework
- 36.
- 37. The Mesos systemhas two basic methods to describe the slaves that comprise a cluster. One of these is managed by the Mesos master, the other is simply passed onwards to the frameworks using the cluster. Attributes The attributes are simply key value string pairs that Mesos passes along when it sends offers to frameworks. Resources & Attributes
- 38. Resources The Mesos systemcan manage 3 different types of resources: scalars, ranges, and sets. These are used to represent the different resources that a Mesos slave has to offer. For example, a scalar resource type could be used to represent the amount of memory on a slave. Each resource is identified by a key string.
- 39. Predefined Uses &Conventions The Mesos master has a few resources that it pre-defines in how it handles them. At the current time, this list consist of: ● ● ● ● In particular, a slave without and resources will never have its resources advertised to any frameworks. Also, the Master’s user interface interprets the scalars in and in terms of . IE: the value is displayed as .
- 40. Examples Here are someexamples for configuring the Mesos slaves. In this case, we have three different types of resources, scalars, a range, and a set. They are called , , , and the range type is . ● scalar called , with the value ● scalar called , with the value ● scalar called , with the value ● range called , with values through (inclusive) ● set called , with the values , and In the case of attributes, we end up with three attributes: ● with value ● with value ● with value
- 41. Roles Total consumable resourcesper slave, in the form 'name(role):value;name(role):value...'. This value can be set to limit resources per role, or to overstate the number of resources that are available to the slave. All * roles will be detected, so you can specify only the resources that are not all roles (*).
- 42. Constraints Constraints control whereapps run to allow optimizing for fault tolerance or locality. Constraints are made up of three parts: a field name, an operator, and an optional parameter. The field can be the slave hostname or any Mesos slave attribute. Fields Hostname field field matches the slave hostnames, see for usage example. field supports all operators of Marathon. Attribute field If the field name is none of the above, it will be treated as a Mesos slave attribute. Mesos slave attribute is a way to tag a slave node, see to learn how to set the attributes.
- 43. Unique tells Marathon toenforce uniqueness of the attribute across all of an app's tasks. For example the following constraint ensures that there is only one app task running on each host: via the Marathon gem: via curl:
- 44. allows you torun all of your app's tasks on slaves that share a certain attribute. You can also use this attribute to tie an application to a specific node by using the hostname property: Cluster
- 45. Group By can beused to distribute tasks evenly across racks or datacenters for high availability. via the Marathon gem: via curl: Optionally, you can specify a minimum number of groups to try and achieve.
- 46. Like accepts a regularexpression as parameter, and allows you to run your tasks only on the slaves whose field values match the regular expression. via the Marathon gem: via curl:
- 47. Unlike Just like operator,but only run tasks on slaves whose field values don't match the regular expression. via the Marathon gem: via curl:
- 48. Go Mesos Constraints& More! https://github.com/elodina/go-mesos-utils --constraints: Constraints (hostname=like:^master$,rack=like:^1.*$).
- 50.