Downloaded 48 times
![Configuration Example config file (zk.conf): Group configuration = Instance + Flavor { "Properties": { /* raw configuration */ } } { "groups" : { "my_zookeeper_nodes" : { "Properties" : { "Instance" : { "Plugin": "instance-vagrant", "Properties": { "Box": "bento/ubuntu-16.04" } }, "Flavor" : { "Plugin": "flavor-zookeeper", "Properties": { "type": "member", "IPs": ["192.168.1.200", "192.168.1.201", "192.168.1.202"] } } } } } }](https://image.slidesharecdn.com/moby-oscon-2017-170509174657/75/Oscon-2017-Build-your-own-container-based-system-with-the-Moby-project-57-2048.jpg)
Uploaded byPatrick Chanezon
Oscon 2017: Build your own container-based system with the Moby project
The document discusses the Moby project and Docker's efforts to build a platform for container-based systems, emphasizing the need for secure, lean, and portable Linux subsystems. It highlights the evolution of Docker and Moby, showcasing their components and collaborative approach to innovate in the container ecosystem. Additionally, it presents Infrakit as a toolkit for declarative infrastructure management, aimed at simplifying and enhancing container orchestration and infrastructure operations.
Oscon 2017: Build your own container-based system with the Moby project
- 1. Patrick Chanezon, @chanezon DavidChung, @dchungsf Mindy Preston, @mindypreston Build your own container-based system with the Moby project May 2017
- 2. French Polyglot Platforms Software Plumber San Francisco DeveloperRelations @chanezon
- 3.
- 4. The world needs toolsof mass innovation
- 5. A programmable Internetwould be the ultimate tool of mass innovation
- 6. A commercial product, builton a development platform, built on infrastructure, built on standards. Docker is building a stack to program the Internet
- 7. Docker is buildinga stack to program the Internet CE EE
- 8. enterprise edition Ubuntu Fedora Mac Azure CentOS Windows 10 AWS Debian communityedition Ubuntu Windows Server Azure CentOS Suse Red Hat AWS Oracle Linux
- 9. Orchestration Container Runtime OS Infrastructure Management ContainerPlatform Layers Application Services
- 10. Docker is aplatform made of components Raft Store Node Identity Secrets Routing Mesh Overlay Networking Swarm Orchestration Engine Application Services
- 11. 12,000,000,000 11,000,000,000 10,000,000,000 9,000,000,000 8,000,000,000 7,000,000,000 6,000,000,000 5,000,000,000 4,000,000,000 3,000,000,000 2,000,000,000 1,000,000,000 Notary runC containerd HyperKit , VPNKit,DataKit SwarmKit libcontainer libnetwork InfraKit 2013 2014 2015 2016 2017 1M 2014 PULLS 1B 2015 PULLS 6B 2016 PULLS 12B 2017 PULLS linuxKit
- 12. LinuxKit A toolkit forbuilding secure, portable and lean operating systems for containers
- 13. Taking Docker multi-platform “I wantDocker for X”
- 14. Desktop Server Cloud Iwant Docker for…
- 15. Not every platform providesa Linux subsystem
- 16. Not every platformprovides a Linux subsystem Orchestration Container Runtime Linux Subsystem Infrastructure Management Application Services
- 17. The container movement needsa secure, lean, portable subsystem
- 18. The container movementneeds a secure, lean, portable Linux subsystem. introducing
- 19. Only works with containers -Smaller attack surface - Immutable infrastructure - Sandboxed system services - Specialized patches and configuration Incubator for security innovations - Wireguard, Landlock, KSPP - MirageOS type safe system daemons Community-first security process - Linux is too big for any one company to secure it - Participate in existing Linux security efforts 1. LinuxKit: a SECURE Linux subsystem
- 20. - Minimal size,minimal boot time - All system services are containers - Everything can be removed or replaced 2. LinuxKit: a LEAN Linux subsystem
- 21. - Desktop, server,IoT, mainframe - Intel & ARM - Bare metal & virtualized 3. LinuxKit: a PORTABLE Linux subsystem
- 22. Docker and Microsoftcollaborate to bring Linux containers to Windows + +
- 23.
- 24. Moby An open frameworkto assemble specialized container systems without reinventing the wheel.
- 25. Pioneers 2013 -2014
- 26.
- 27. Use case: cloudnative apps on Linux server Early Adopters 2015 - 2016
- 28. Production Model: OPENCOMPONENTS
- 29. Mainstream 2017 -2018 Containers are spreading to every category of computing: server, datacenter, cloud, IoT, desktop, mobile…
- 30. Case study: Specializing Dockerfor the mainstream Desktop Server Cloud
- 31. The open componentmodel shows its limits…
- 32. The auto industryhas solved this problem: COMMON ASSEMBLIES.
- 33. Scaling the Dockerproduction model: share components AND ASSEMBLIES.
- 34. It’s time totake our ecosystem to the next level… By collaborating on components AND COMMON ASSEMBLIES.
- 36. – Library of80+ components – Package your own components as containers – Reference assemblies deployed on millions of nodes – Create your own assemblies or start from an existing one A framework to assemble specialized container systems without reinventing the wheel.
- 37. Docker uses Mobyfor its open-source – Thousands of contributors, hundreds of patches/week – Component development – Specialized assembly development – Integration tests – Architecture design – Integration with other projects – Experimentation and bleeding edge features
- 38. Docker uses Mobyfor its open-source... and so can you! – Community-run – Open governance inspired by the Fedora project – Plays well with existing projects - no donation necessary!
- 39.
- 40. What it meansfor you Moby helps you innovate without tying you to Docker System BuildersDocker Users Docker will better leverage the ecosystem to innovate faster for you
- 41. Moby transforms multi-monthR&D projects into weekend projects.
- 42.
- 43. custom CI/CD stack Weekend project #2: NotaryRegistry Docker Builder +
- 44. custom CI/CD stack + Debian +Terraform Weekend project #3: Notary Docker Builder + Registry
- 45.
- 46. "RedisOS" for Windows "RedisOS" for Mac "RedisOS" forbare metal HyperKit bare metal
- 47. Etcd clustering on GoogleCloud Weekend project #5:
- 48. SSHD Kubernetes on the Mac Weekend project#6: HyperKit
- 49. Getting Started - Bloghttps://mobyproject.org/blog - Twitter @moby - Github moby/moby
- 50. Let’s take containersmainstream!
- 52. InfraKit A toolkit forbuilding declarative, self-healing infrastructure.
- 53. What is it? 53 •Launched at LinuxCon, Berlin in October, 2016. • Toolkit for building declarative, self-managing distributed applications • Active management with active controllers • scaling groups, rolling updates • monitoring / health checks • connecting nodes to L4 / ingress • Declarative infrastructure
- 54.
- 55. container orchestration Where doesit fit? 55 kubectl run nginx --image=nginx gcloud container node-pools list --zone us- central1-f --cluster MyWorkers aws autoscaling update-auto-scaling-group --auto-scaling-group-name MyWorkers docker create service nginx … infrakit group describe workers az vmss create --resource-group vmss- test-1 --name MyWorkers container orchestration infrastructure orchestrationinfrastructure orchestration list, err := group.Controller.Describe(“workers”)
- 56. App Opscontainer orchestrationAppOps One console across environments 56 kubectl run nginx --image=nginx docker create service nginx … infrakit group describe workers container orchestration infrastructure orchestration list, err := group.Controller.Describe(“workers”) AWS RackHDAZ GCP OneVIEWMAASKVM VMW Cloud Ops Hardware OpsCluster Ops
- 57. Configuration Example config file(zk.conf): Group configuration = Instance + Flavor { "Properties": { /* raw configuration */ } } { "groups" : { "my_zookeeper_nodes" : { "Properties" : { "Instance" : { "Plugin": "instance-vagrant", "Properties": { "Box": "bento/ubuntu-16.04" } }, "Flavor" : { "Plugin": "flavor-zookeeper", "Properties": { "type": "member", "IPs": ["192.168.1.200", "192.168.1.201", "192.168.1.202"] } } } } } }
- 58.
- 59. Support more platforms 59 •Compute: • Bare-metal: HP OneView, MAAS, RackHD • Public cloud: AWS, GCP • MacOS X (HyperKit); Docker containers • Coming soon: Azure, IBM, Digital Ocean, Packet, libvirt • Other resource types • AWS - vpc, subnets, gateways, etc.
- 60. Improve usability 60 • Templates •Complex scripts and configuration in any format; no more escape quotes in JSON • Fetch templates from remote repositories • Playbooks • CLI - flags, prompts — config driven and dynamic • Share “playbooks” from remote repositories
- 61. Improve core system 61 •High Availability — Swarm Mode or etcd • New Plugin types — Metadata and Events • Metadata: cluster-wide sysfs and reflection • Events - publish / subscribe • Remote client access: infrakit -H host:port to remote cluster
- 62.
- 63. Use Cases 63 • Supportcontainer orchestration • bootstrapping + day N management • API for cluster autoscaling • k8s, Docker Swarm Mode • Bare-metal + GPU provisioning • IoT — LinuxKit integration / custom kernel deployment
- 64. Improve usability 64 • FinalizeAPI / Schema for 1.0 • Make it easy to consume • Simplify setup - fewer daemons and binaries • Embeddable / vendor API • Sensible CLI for stable / experimental features • Make it easy to extend / contribute • metadata / instance plugins • playbooks / reusable templates • community CI / compatibility testing • Documentation
- 65. Improve core system 65 •Provisioning of diverse resource types • networks / proxies / load balancers • GPU • Stability / performance of core controllers • Asynchronous messaging - mqtt, natsd, amqp • Monitoring + Health check SPI
- 66. Support more platforms 66 •Direct libvirt / KVM / CUDA • Better bare-metal / hardware ops integration • Kernel image build pipeline — LinuxKit Build, test, and deploy clusters from infrastructure definitions to kernel images
- 67.
- 69. Learn More atOSCON - Mindy Preston, Amir Chaudhry’s “MirageOS 3: Smaller, lighter, and more transparent” Wednesday 4:15 pm - David Chung, Bill Farner “InfraKit: A toolkit for infrastructure orchestration” Thursday 11 am
- 70.
Editor's Notes
- #9 Docker has its roots in dev productivity still lot of work to do solve it by listening to devs, solving all their pbs one by one let’s talk about dev problems.
- #10 our job is to give you the best tools to take your app to prod securely 3 rules we follow usability, portability, scalability usable security so that devs don't bypass it scale, automation Docker suite of tools for security in production tools deliver security that is usable scalable portable not getting in the way of operators help developers make applications be more secure has to be easy and portable so that developers will use it. Security never ends, continuous process We made a lot of progress this year 4 features I want to talk about
- #11 reason that Docker is so componentized because of open dev model we adopted partnered with an ecosystem that grew around it several phases ---- docker is a container platform solve pb for our users develop new components, or improve existing components open dev model, 12 oss projects produce one comp of a container platform any one project useless on its own
- #14 one story how we used this model to solve toughest tech pb way could not have without it moby at docker to innovate faster editions case study of using moby pb: going beyond linux no need to bring you own linux os originally Linux only market we want to use docker everywhere we don't have a favorite linux product we want to use containers our platform is not linux mac, windows 10, azure, aws, gcp portable container platform, need docker to work on our existing platform
- #15 one story how we used this model to solve toughest tech pb way could not have without it moby at docker to innovate faster editions case study of using moby pb: going beyond linux no need to bring you own linux os originally Linux only market we want to use docker everywhere we don't have a favorite linux product we want to use containers our platform is not linux mac, windows 10, azure, aws, gcp portable container platform, need docker to work on our existing platform
- #16 Docker has its roots in dev productivity still lot of work to do solve it by listening to devs, solving all their pbs one by one let’s talk about dev problems.
- #18 Docker has its roots in dev productivity still lot of work to do solve it by listening to devs, solving all their pbs one by one let’s talk about dev problems.
- #19 we partnerd with several companies and LF to create this component ongoing efforts to port it to diff environments, map of different efforts HPE VMW Mainframe IoT ARM Windows Clouds….
- #20 everything uses containers, every major Linux based project has embraced containers containers are how Linux will work going fwd Based on containerd System services running as containers LinuxKit is the place to try out new Linux security ideas Provably secure, but also Option to be at the bleeding edge of security secure edge channel, key feature of the project Working with industry to make LinuxKit the most secure OS for everybody
- #22 Base for Linux functionality on any environment ex: GE EdgeOS, exotic iot devices, cannot do that with an OS designed for Datacenter Ways to make Linux secure, but locked into a specific environment Take advantage of every aspect of infrastructure
- #23 Microsoft partnering with Docker to bring Linux to Windows
- #26 bleeding edge cloud developers
- #29 define components of the platform as containers allows more advanced production models multiple stages of collaboration
- #30 deploying at scale in specialized systems
- #31 last year editions, mac, windows, cloud
- #32 how do we deliver all editions of our platform and scale our engineering single systemn company to multi system company each is very specialized whole point of docker is to deliver a standard experience using containers
- #33 steal an idea from auto industry assemblies, different ways of assembling components open common assemblies for iot, mainframe, cloud native, desktop achieve scale through openness propose ecosystem to share this model of growth component and assembly level, everyone can bring their components and assemblies and innovate at both levels
- #34 introduced assemblies allowed docker to scale internally allowed us to ramp up to 12 editions of docker complexity duplication of effort, design allowed docker to scale internally allowed us to ramp up to 12 editions of docker introduced an additional level of collaboration, assembly, cpatures what is common and that teams can use for their environments
- #35 looks like this, moby origin, assembly we use to create editions of docker all of our assembly dev will take place in moby project if want close derivative to docker platform, join dev of moby origin want diofferent assembly, fork moby-origin partners Introducing a new project where Docker does 100% of its oss work
- #37 all components all assemblies inviting users, partners the whole ecosystem to join this project and together take container ecosystem to the mainstream seed this project with 12s of components, an assembly that is very stable and deployed in prod on 1000s nodes the most important project we have introduced since 2014 not a foundation moby does not own projects any project can come collab and retain ownership of their code
- #38 all components all assemblies inviting users, partners the whole ecosystem to join this project and together take container ecosystem to the mainstream seed this project with 12s of components, an assembly that is very stable and deployed in prod on 1000s nodes the most important project we have introduced since 2014 not a foundation moby does not own projects any project can come collab and retain ownership of their code
- #39 all components all assemblies inviting users, partners the whole ecosystem to join this project and together take container ecosystem to the mainstream seed this project with 12s of components, an assembly that is very stable and deployed in prod on 1000s nodes the most important project we have introduced since 2014 not a foundation moby does not own projects any project can come collab and retain ownership of their code
- #40 platform based on containers
- #41 Docker uses Moby to innovate in the open. Each version of Docker will innovate faster Moore innovation/more choice
- #42 all components all assemblies inviting users, partners the whole ecosystem to join this project and together take container ecosystem to the mainstream seed this project with 12s of components, an assembly that is very stable and deployed in prod on 1000s nodes the most important project we have introduced since 2014 not a foundation moby does not own projects any project can come collab and retain ownership of their code
- #50 platform based on containers
- #51 platform based on containers
- #55 InfraKit can be used set up to run with leader election (e.g. integrated with Docker swarm mode) to achieve high availability. Multiple plugin sets are running as hot standby but only one is active (the leader).
- #58 Simple patterns Compose them into a large configuration. Group is made up of Instance and Flavor