At Devoxx Belgium and Devoxx Morocco, Ray Tsang and I (Arjen Wassink) showed a Raspberry Pi cluster we built at Quintor running HypriotOS, Docker and Kubernetes. While we received many compliments on the talk, the most common question was about how to build a Pi cluster themselves! We’ll be doing just that, in two parts. The first part covered the shopping list for the cluster, and this second one will show you how to get kubernetes up and running . . .

Installing the Kubernetes master node

• docker-bootstrap.service - is a separate Docker daemon to run etcd and flannel because flannel needs to be running before the standard Docker daemon (docker.service) because of network configuration.
• k8s-etcd.service - is the etcd service for storing flannel and kubelet data.
• k8s-flannel.service - is the flannel process providing an overlay network over all nodes in the cluster.
• docker.service - is the standard Docker daemon, but with flannel as a network bridge. It will run all Docker containers.
• k8s-master.service - is the kubernetes master service providing the cluster functionality.

Deploying the first pod and service on the cluster

Installing the Kubernetes worker nodes

Enjoy your Kubernetes cluster!

Overview 

Implementing leader election in Kubernetes 

• ResourceVersions - Every API object has a unique ResourceVersion, and you can use these versions to perform compare-and-swap on Kubernetes objects 
• Annotations - Every API object can be annotated with arbitrary key/value pairs to be used by clients. 

$ kubectl run leader-elector --image=gcr.io/google_containers/leader-elector:0.4 --replicas=3 -- --election=example

$ kubectl get pods
NAME                   READY     STATUS    RESTARTS   AGE
leader-elector-inmr1   1/1       Running   0          13s
leader-elector-qkq00   1/1       Running   0          13s
leader-elector-sgwcq   1/1       Running   0          13s

${pod_name}, (e.g. leader-elector-inmr1 from the above)

$ kubectl logs -f ${name}
leader is (leader-pod-name)

# ‘example’ is the name of the candidate set from the above kubectl run … command
$ kubectl get endpoints example -o yaml

$ kubectl delete pods (leader-pod-name)

# delete the old leader elector group
$ kubectl delete rc leader-elector

# create the new group, note the --http=localhost:4040 flag
$ kubectl run leader-elector --image=gcr.io/google_containers/leader-elector:0.4 --replicas=3 -- --election=example --http=0.0.0.0:4040

# create a proxy to your Kubernetes api server
$ kubectl proxy

http://localhost:8001/api/v1/proxy/namespaces/default/pods/(leader-pod-name):4040/

And you will see:

{"name":"(name-of-leader-here)"}

Leader election with sidecars 

var http = require('http');
// This will hold info about the current master
var master = {};

// The web handler for our nodejs application
var handleRequest = function(request, response) {
    response.writeHead(200);
    response.end("Master is " + master.name);
};

// A callback that is used for our outgoing client requests to the sidecar
var cb = function(response) {
    var data = '';
    response.on('data', function(piece) { data = data + piece; });
    response.on('end', function() { master = JSON.parse(data); });
};

// Make an async request to the sidecar at http://localhost:4040
var updateMaster = function() {
    var req = http.get({host: 'localhost', path: '/', port: 4040}, cb);
    req.on('error', function(e) { console.log('problem with request: ' + e.message); });
    req.end();
};

// Set up regular updates
updateMaster();
setInterval(updateMaster, 5000);

// set up the web server
var www = http.createServer(handleRequest);
www.listen(8080);

Conclusion 

• Note taker: Joe Beda
• Demonstration: Automated Deploy on Metal, AWS and others w/ Digital Rebar, Rob Hirschfeld  and Greg Althaus from RackN
• Greg Althaus. CTO.  Digital Rebar is the product.  Bare metal provisioning tool.
• Detect hardware, bring it up, configure raid, OS and get workload deployed.
• Been working on Kubernetes workload.
• Seeing trend to start in cloud and then move back to bare metal.
• New provider model to use provisioning system on both cloud and bare metal.
• UI, REST API, CLI
• Demo: Packet -- bare metal as a service
• 4 nodes running grouped into a “deployment”
• Functional roles/operations selected per node.
• Decomposed the kubernetes bring up into units that can be ordered and synchronized.  Dependency tree -- things like wait for etcd to be up before starting k8s master.
• Using the Ansible playbook under the covers.
• Demo brings up 5 more nodes -- packet will build those nodes
• Pulled out basic parameters from the ansible playbook.  Things like the network config, dns set up, etc.
• Flannel now with work on opencontrail
• Hierarchy of roles pulls in other components -- making a node a master brings in a bunch of other roles that are necessary for that.
• Has all of this combined into a command line tool with a simple config file.
• Forward: extending across multiple clouds for test deployments.  Also looking to create split/replicated across bare metal and cloud.
• Q: secrets?
  A: using ansible playbooks.  Builds own certs and then distributes them.  Wants to abstract them out and push that stuff upstream.
• Q: Do you support bringing up from real bare metal with PXE boot?
  A: yes -- will discover bare metal systems and install OS, install ssh keys, build networking, etc.
• [from SIG-scalability] Q: What is the status of moving to golang 1.5?
  A: At HEAD we are 1.5 but will support 1.4 also. Some issues with flakiness but looks like things are stable now.  
• Also looking to use the 1.5 vendor experiment.  Move away from godep.  But can’t do that until 1.5 is the baseline.
• Sarah: one of the things we are working on is rewards for doing stuff like this.  Cloud credits, tshirts, poker chips, ponies.
• [from SIG-scalability] Q: What is the status of cleaning up the jenkins based submit queue? What can the community do to help out?
  A: It has been rocky the last few days.  There should be issues associated with each of these. There is a flake label on those issues.  
• Still working on test federation.  More test resources now.  Happening slowly but hopefully faster as new people come up to speed.  Will be great to having lots of folks doing e2e tests on their environments.
• Erick Fjeta is the new test lead
• Brendan is happy to help share details on Jenkins set up but that shouldn’t be necessary.
• Federation may use Jenkins API but doesn’t require Jenkins itself.
• Joe bitches about the fact that running the e2e tests in the way Jenkins is tricky.  Brendan says it should be runnable easily.  Joe will take another look.
• Conformance tests? etune did this but he isn’t here.  - revisit 20150121
• Kubecon.io/EU CFP is open https://kubecon.io/call-for-proposals/
• March 10-11 in London.  Venue to be announced this week.
• Please send talks!  CFP deadline looks to be Feb 5.
• Would love to see more talks from production users.
• Lots of excitement.  Looks to be 700-800 people.  Bigger than SF version (560 ppl).
• Buy tickets early -- early bird prices will end soon and price will go up 100 GBP.
• Accommodations provided for speakers?
• Potentially but not 100% certain.  Need to figure it out.
• Q from Bob @ Samsung: Can we get more warning/planning for stuff like this:
• A: Sarah -- I don’t hear about this stuff much in advance but will try to pull together a list.  Working to make the events page on kubernetes.io easier to use.
• A: JJ -- we’ll make sure we give more info earlier for the next US conf.
• Scale tests [Rob Hirschfeld from RackN] -- if you want to help coordinate on scale tests we’d love to help.
• Bob invited Rob to join the SIG-scale group.
• There is also a big bare metal cluster through the CNCF (from Intel) that will be useful too.  No hard dates yet on that.
• Notes/video going to be posted on k8s blog. (Video for 20150114 wasn’t recorded.  Fail.)

January 21 - Configuration, Federation and Testing, oh my. 

• • Note taker: Rob Hirshfeld
  • Use Case (10 min): SFDC Paul Brown
  • SIG Report - SIG-config and the story of #18215
  • ApplicationconfigINK8s,notdeploymentofK8s
  • Topichasbeenreuseofconfiguration,specificallyparameterization(akatemplates). Needs:
  • needsincludescoping(clusternamespace)
  • slightcustomization (naming changes, but not major config)
  • multiplepositionsonhowtodothisincluding
  • allowingexternalorsimpleextensions.
  • PetSetcreatesinstancesw/stablenamespace
  • Workflowproposal
  • DistributedChron.
  • Challengeisthatconfigsneedtocreatemultipleobjectsinsequence
  • Tryingtofigureouthowbalancethemanyconfigoptionsoutthere(compose,terraform,ansible/etc)
  • Goalisto“meetpeoplewheretheyare”tokeepitsimple
  • Q: is there an opinion for the keystore sizing
  • large size / data blob would not be appropriate
  • you can pull data(config) from another store for larger objects
  • SIG Report - SIG-federation - progress on Ubernetes-Lite & Ubernetes design
  • Goalittobeabletohaveaclustermanager,soyoucanfederateclusters.  They will automatically distribute the pods.
  • PlanistousethesameAPIforthemastercluster
  • Quinton's Ubernetes Talk:  https://youtu.be/L2ZK24JojB4
  • Design for Ubernetes: https://github.com/kubernetes/kubernetes/pull/19313
  
  
  • Conformance testing Q+A [Isaac Hollander McCreery]
  • status on conformance testing for release process
  • expect to be forward compatible but not backwards
  • is there interest for a sig-testing meeting
  • testing needs to a higher priority for the project
  • lots of focus on trying to make this a higher priority

Now, without further ado, the data:

Development Complexity

• “Silo'd development environments / workflows can be fragmented, ease of access to tools like logs is available when debugging containers but not intuitive at times, massive amounts of knowledge is required to grasp the whole infrastructure stack and best practices from say deploying / updating kubernetes, to underlying networking etc.”
• “Migrating developer workflow. People uninitiated with containers, volumes, etc just want to work.”

Security

• “Network Security”
• “Secrets”

Immaturity

• “Lack of a comprehensive non-proprietary standard (i.e. non-Docker) like e.g runC / OCI”
• “Still early stage with few tools and many missing features.”
• “Poor CI support, a lot of tooling still in very early days.”
• "We've never done it that way before."

Complexity

• “Networking support, providing ip per pod on bare metal for kubernetes”
• “Clustering is still too hard”
• “Setting up Mesos and Kubernetes too damn complicated!!”

Data

• “Lack of flexibility of volumes (which is the same problem with VMs, physical hardware, etc)”
• “Persistency”
• “Storage”
• “Persistent Data”

January 28 - 1.2 release update, Deis demo, flaky test surge and SIGs

• Note taker: Erin Boyd
• Discuss process around code freeze/code slush (TJ Goltermann)
• Code wind down was happening during holiday (for 1.1)
• Releasing ~ every 3 months
• Build stability is still missing
• Issue on Transparency (Bob Wise)
• Email from Sarah for call to contribute (Monday, January 25)
• Concern over publishing dates / understanding release schedule /etc…
• Release targeted for early March
• Where does one find information on the release schedule with the committed features?
• For 1.2 - Send email / Slack to TJ
• For 1.3 - Working on better process to communicate to the community
• Twitter
• Wiki
• GitHub Milestones
• How to better communicate issues discovered in the SIG
• AI: People need to email the kubernetes-dev@ mailing list with summary of findings
• AI: Each SIG needs a note taker
• Release planning vs Release testing
• Testing SIG lead Ike McCreery
• Also part of the testing infrastructure team at Google
• Community being able to integrate into the testing framework
• Federated testing
• Release Manager = David McMahon
• Request to  introduce him to the community meeting
• Demo: Jason Hansen Deis
• Implemented simple REST API to interact with the platform
• Deis managed application (deployed via)
• Source -> image
• Rolling upgrades -> Rollbacks
• AI: Jason will provide the slides & notes
• Slides: https://speakerdeck.com/slack/kubernetes-community-meeting-demo-january-28th-2016
• Alpha information: https://groups.google.com/forum/#!topic/deis-users/Qhia4DD2pv4
• Adding an administrative component (dashboard)
• Helm wraps kubectl
• Testing
• Called for community interaction
• Need to understand friction points from community
• Better documentation
• Better communication on how things “should work"
• Internally, Google is having daily calls to resolve test flakes
• Started up SIG testing meetings (Tuesday at 10:30 am PT)
• Everyone wants it, but no one want to pony up the time to make it happen
• Google is dedicating headcount to it (3-4 people, possibly more)
• https://groups.google.com/forum/?hl=en#!forum/kubernetes-sig-testing
• Best practices for labeling
• Are there tools built on top of these to leverage
• AI: Generate artifact for labels and what they do (Create doc)
• Help Wanted Label - good for new community members
• Classify labels for team and area
• User experience, test infrastructure, etc..
• SIG Config (not about deployment)
• Any interest in ansible, etc.. type
• SIG Scale meeting (Bob Wise & Tim StClair)
• Tests related to performance SLA get relaxed in order to get the tests to pass
• exposed process issues
• AI: outline of a proposal for a notice policy if things are being changed that are critical to the system (Bob Wise/Samsung)
• Create a Best Practices of set of constants into well documented place

February 4th - rkt demo (congratulations on the 1.0, CoreOS!), eBay puts k8s on Openstack and considers Openstack on k8s, SIGs, and flaky test surge makes progress.

• Note taker: Rob Hirschfeld
• Demo (20 min): CoreOS rkt + Kubernetes [Shaya Potter]
• rkt 1.0 released today
• expect to see integrations w/ rkt & k8s in the coming months (“rkt-netes”). not integrated into the v1.2 release.
• Shaya gave a demo (8 minutes into meeting for video reference)
• CLI of rkt shown spinning up containers
• [note: audio is garbled at points]
• Discussion about integration w/ k8s & rkt
• rkt community sync next week: https://groups.google.com/forum/#!topic/rkt-dev/FlwZVIEJGbY
• Dawn Chen:
• The remaining issues of integrating rkt with kubernetes: 1) cadivsor 2) DNS 3) bugs related to logging
• But need more work on e2e test suites
• Use Case (10 min): eBay k8s on OpenStack and OpenStack on k8s [Ashwin Raveendran]
• eBay is currently running Kubernetes on OpenStack
• Goal for eBay is to manage the OpenStack control plane w/ k8s.  Goal would be to achieve upgrades
• OpenStack Kolla creates containers for the control plane.  Uses Ansible+Docker for management of the containers.  
• Working on k8s control plan management - Saltstack is proving to be a management challenge at the scale they want to operate.  Looking for automated management of the k8s control plane.
• SIG Report
• Wikipage: https://github.com/kubernetes/kubernetes/wiki/Special-Interest-Groups-(SIGs)
• updated by Joe Beda.  Please add SIGs there.
• SIG calendar
• SIG Openstack [Ashwin Raveendran]
• Meets Tuesday afternoon, bi-weekly
• Kubernetes-Sig-Openstack
• Launch SIG Cluster Ops or SIG Deploy.  [Rob Hirschfeld].  please review Draft Charter
• working to standardize operationalization of Kubernetes
• Testing update [Jeff, Joe, and Erick]
• Working to make the workflow about contributing to K8s easier to understanding
• pull/19714 has flow chart of the bot flow to help users understand
• Need a consistent way to run tests w/ hacking config scripts (you have to fake a Jenkins process right now)
• Want to create necessary infrastructure to make test setup less flaky
• want to decouple test start (single or full) from Jenkins
• goal is to get to point where you have 1 script to run that can be pointed to any cluster
• demo included Google internal views - working to try get that external.
• want to be able to collect test run results
• Bob Wise calls for testing infrastructure to be a blocker on v1.3
• Long discussion about testing practices…
• consensus that we want to have tests work over multiple platforms.
• would be helpful to have a comprehensive state dump for test reports
• “phone-home” to collect stack traces - should be available
• 1.2 Release Watch
• it’s coming!!!
• committed in 1.2
• might make 1.2
• code slush starts - Friday or Monday
• CoC [Sarah]
• =========== RAN OUT OF TIME ===============
• Existing CoC: https://github.com/kubernetes/kubernetes/blob/master/code-of-conduct.md
• Previous Discussion: https://github.com/kubernetes/kubernetes/pull/13578
• GSoC [Sarah]
• need mentors!!  deadline is very soon.

TL;DRKubernetes has been a key component for us to reduce technical debt in our infrastructure by:

• Fostering the Adoption of Docker
• Simplifying Container Management
• Onboarding Developers On Infrastructure
• Unlocking Continuous Integration and Delivery

The Problem

The Solution

Kubernetes As A Tool To Foster Docker Adoption

Kubernetes As A Tool To Manage Containers

• It is easy to install on AWS (where all our apps were running)
• There is a direct path from a Dockerfile to a replication controller through a yaml/json file
• Pods are able to scale in number easily
• We can easily scale the number of VM’s running on AWS in a Kubernetes cluster
• Rolling deployments and rollback are built into the tooling
• Each pod gets monitored through health checks
• Service endpoints are managed by the tool
• There is an active and vibrant community

Kubernetes As A Tool For Onboarding Developers On Infrastructure

Kubernetes As A Means To Unlock Continuous Integration And Delivery

Next Steps

Summary

February 11th - Pangaea Demo, #AWS SIG formed, release automation and documentation team introductions. 1.2 update and planning 1.3. 

• Note taker: Rob Hirschfeld
• Demo: Pangaea [Shahidh K Muhammed, Tanmai Gopal, and Akshaya Acharya]
• Microservices packages
• Focused on Application developers
• Demo at recording +4 minutes
• Single node kubernetes cluster — runs locally using Vagrant CoreOS image
• Single user/system cluster allows use of DNS integration (unlike Compose)
• prevents collisions
• Can run locally or in cloud
• Best contact is via the Pangaea repo
• SIG Report
• SIG Cluster Ops
• collecting operators and tools (inventory)
• will set time for meeting:  recommendations?  time zone?
• Hello world from SIG-AWS
• running K8s on AWS
• Release Automation and an introduction to David McMahon
• Current is heavily documented but not automated
• objectives
• separate build and release
• make it more of a software process (less manual & more repeatable)
• target to have framework built (automation will not be ready next release)
• Docs and k8s website redesign proposal and an introduction to John Mulhausen
• Switching from native script munging to Jekyll (http://jekyllrb.com/docs/home/)
• This will allow the system to build docs correctly from Github w/ minimal effort
• Will be check-in triggered
• Getting website style updates
• Want to keep authoring really light
• There will be some automated checks
• Next week: preview of the new website during the community meeting
• [@goltermann] 1.2 Release Watch (time +34 minutes)
• committed in 1.2
• might make 1.2
• code slush date: 2/9/2016
• not 100% but close.  some resources still moving from beta to v1
• no major features or refactors accepted
• discussion about release criteria: we will hold release date for bugs
• we’re getting accurate counts of bugs.  
• hard to predict burn down at this point
• Testing flake surge is over (one time event and then maintain test stability)
• if you find a “flaky” test, then it’s a P0 to fix it.  Want to eliminate false fail test results
• they are down by 75%!  (meaning, that 75% of them are eliminated)
• 1.3 Planning (time +40 minutes)
• working to cleanup the Github milestones — they should be a source of truth.  you can use Github for bug reporting
• push off discussion while 1.2 crunch is under
• Framework
• dates
• prioritization
• feedback
• Design Review meetings
• General discussion about the PRD process — still at the beginning states

• Working on a contributor conference
• Rob suggested tracking relationships between PRD/Mgmr authors
• PLEASE DO REVIEWS — talked about the way people are authorized to +2 reviews.

February 18th - kmachine demo, clusterops SIG formed, new k8s.io website preview, 1.2 update and planning 1.3

• Note taker: Rob HIrschfeld
• Demo (10 min): kmachine [Sebastien Goasguen]
• started :01 intro video
• looking to create mirror of Docker tools for Kubernetes (similar to machine, compose, etc)
• kmachine (forked from Docker Machine, so has the same endpoints)
• creates a single machine w/ kubernetes endpoint setup
• demo showing AWS & Virtual Box
• Use Case (10 min): started at :15
• Network isolation for different namespaces
• Pods in namespace1 should not be able to contact Pods in namespace2
• No native implementation at this point
• Project Calico and Openshift have plugins that enable this
• Calico implements this at the moment here: https://github.com/projectcalico/calico-containers/blob/master/docs/cni/kubernetes/Policy.md
• More details from Networking SIG
• SIG Report starter
• Cluster Ops launch meeting Friday (doc). [Rob Hirschfeld]
• Time Zone Discussion [:22]
• This timezone does not work for Asia.  
• Considering rotation - once per month
• Likely 5 or 6 PT
• Rob suggested moving the regular meeting up a little
• k8s.io website preview [John Mulhausen] [:27]
• using github for docs.  you can fork and do a pull request against the site
• will be it’s own kubernetes organization BUT not in the code repo
• Google will offer a “doc bounty” where you can get GCP credits for working on docs
• Uses Jekyll to generate the site (e.g. the ToC)
• Principle will be to 100% GitHub Pages; no script trickery or plugins, just fork/clone, edit, and push
• Hope to launch at Kubecon EU
• Source Repo: https://github.com/kubernetes/kubernetes.github.io
• Preview: http://kubernetes.github.io
• Home Page Only Preview: http://kub.unitedcreations.xyz
• 1.2 Release Watch [T.J. Goltermann] [:38]
• committed in 1.2
• might make 1.2
• Discussion - trying to find the balance between early and late branch
• it is likely too early to branch (there are >100 issues)
• thinking to delay by a week based on bug burn down
• for now, we’re leaving it open.  it’s at least a week out
• 1.3 Planning update [T.J. Goltermann]
• google 1.3 feature list to be presented 3/3
• Community members (company or otherwise) commitments requested 3/17?
• GSoC participation -- deadline 2/19  [Sarah Novotny]
• if you’re interested in being a mentor, please reach out  to Sarah today.
• March 10th meeting? [Sarah Novotny]
• March 10th is KubeCon EU.  I’m inclined to cancel the community meeting.  Thoughts?

• “Kubernetes Hardware Hacks: Exploring the Kubernetes API Through Knobs, Faders, and Sliders” by Ian Lewis and Brian Dorsey, Developer Advocate, Google -- http://sched.co/6Bl3
  
• “rktnetes: what's new with container runtimes and Kubernetes” by Jonathan Boulle, Developer and Team Lead at CoreOS -- http://sched.co/6BY7

• “Kubernetes Documentation: Contributing, fixing issues, collecting bounties” by John Mulhausen, Lead Technical Writer, Google -- http://sched.co/6BUP 
• “What is OpenStack's role in a Kubernetes world?” By Thierry Carrez, Director of Engineering, OpenStack Foundation -- http://sched.co/6BYC
• “A Practical Guide to Container Scheduling” by Mandy Waite, Developer Advocate, Google -- http://sched.co/6BZa
  
• “Kubernetes in Production in The New York Times newsroom” Eric Lewis, Web Developer, New York Times -- http://sched.co/67f2
• “Creating an Advanced Load Balancing Solution for Kubernetes with NGINX” by Andrew Hutchings, Technical Product Manager, NGINX -- http://sched.co/6Bc9
• And many more http://kubeconeurope2016.sched.org/

February 25th - Redspread demo, 1.2 update and planning 1.3, newbie introductions, SIG-networking and a shout out to CoreOS blog post.

• Note taker: [Ilan Rabinovich]
• Quick call out for sharing presentations/slides [JBeda]
• https://github.com/jbeda/slides-kubernetes-101
• Demo (10 min):Redspread [Mackenzie Burnett, Dan Gillespie]
• Just open sourced
• YC company
• https://github.com/redspread/spread
• Streamline tool to build/push/deploy to k8s in one command
• Looking forward to offline development of k8s cluster.
• Client only
• reuses a lot of kubectl code
• Convention on directory structure
• Roadmap
• Linking between Kube objects as an app definition.
• Parameterization and versioning of configs (eg git for k8s)
• Happily welcoming contributors and user feedback via github (https://github.com/redspread/spread or firstname@redspread.com)
• Q/A
• Brian Grant asked for feedback on how to reorganize the kubectl code base to make projects like redspread easier.
• 1.2 Release Watch [T.J. Goltermann]
• committed in 1.2
• currently about 80 issues in the queue that need to be addressed before branching.
• currently looks like March 7th may slip to later in the week, but up in the air until flakey tests are resolved.
• non-1.2 changes may be delayed in review/merging until 1.2 stabilization work completes.
• might make 1.2
• 1.3 release planning
• March 17th meeting will discuss features from community members and Google. Bring your notes/plans to that meeting
• Newbie Introductions
• SIG Reports -
• Networking [Tim Hockin]
• Meets later today.
• Working on a (proto)-specification for a 3rd party resource to describe network policies. (eg pod x can talk to service y, or frontends can/cannot talk to backends)
• Currently ironing out naming/structure
• Calico has a running demo that shows Calico project enforcing network policy based on an earlier form of the spec.
• Shouldn’t require any changes to the 1.2 code
• Goal is to submit it as part of the 1.3 cycle.
• Scale [Bob Wise]
• CoreOS Blog post on scheduler scaling- https://coreos.com/blog/improving-kubernetes-scheduler-performance.html
• Cluster Ops [Rob Hirschfeld]
• meeting last Friday went very well.  Discussed charter AND a working deployment
• moved meeting to Thursdays @ 1 (so in 3 hours!)
• Rob is posting a Cluster Ops announce on TheNewStack to recruit more members
• GSoC participation -- no application submitted.  [Sarah Novotny]
• Brian Grant has offered to review PRs that need attention for 1.2
• Dynamic Provisioning
• Currently overlaps a bit with the ubernetes work
• PR in progress.
• Should work in 1.2, but being targeted more in 1.3
• Next meeting is March 3rd.
• Demo from Weave on Kubernetes Anywhere
• Another Kubernetes 1.2 update
• Update from CNCF update
• 1.3 commitments from google
• No meeting on March 10th.  
• See you at Kubecon EU

Containers continue to gain ground

• Development: 80% -> 88%
• Test: 67% -> 72%
• Pre production: 41% -> 55%
• Production: 50% -> 62%

Container and cluster sizes

Things stay the same

Raw data

Conclusions

• Decouple applications from infrastructure - Focus on where the value for the customer is: the application.
• Decompose applications - Build applications from smaller, loosely coupled parts. Enable reconfiguration of those parts depending on the needs of the business. Also encourage innovation by low-cost experiments.
• Automate everything - Fight the increasing complexity of the first two points by introducing a high degree of automation.

• On-premise deployments (with the option for hybrid scenarios)
• Efficient operations as part of a (much) bigger IT infrastructure
• Enterprise-grade support, potentially on global scale

• Proven setup process, lowers risk of problems while setting up the cluster
• Reduction of provisioning time to minutes
• Repeatable process, relevant especially for large, multi-tenant environments

• Platform-oriented, relates to the overall infrastructure, often including various systems, one of which might be Kubernetes.
• Application-oriented, focusses rather on a single, or a small set of applications deployed on Kubernetes.

• Configure Google Container Engine (GKE) for cluster installation and management

• Use Kubernetes to provision the infrastructure and clusters for containers  

• Use your existing tools of choice to actually build your containers
• Use ElasticKube to run, deploy and manage your containers and services

1. Deploy ElasticKube to a Kubernetes cluster
2. Configuration
3. Admin: Setup and invite a user
4. Deploy an instance

What’s new: 

• Significant scale improvements. Increased cluster scale by 400% to 1,000 nodes and 30,000 containers per cluster.
• Simplified application deployment and management. 
• Dynamic Configuration (via the ConfigMap API) enables applications to pull their configuration when they run rather than packaging it in at build time. 
• Turnkey Deployments (via the Beta Deployment API) let you declare your application and Kubernetes will do the rest. It handles versioning, multiple simultaneous rollouts, aggregating status across all pods, maintaining application availability and rollback. 
• Automated cluster management:
• Improved reliability through cross-zone failover and multi-zone scheduling
• Simplified One-Pod-Per-Node Applications (via the Beta DaemonSet API) allows you to schedule a service (such as a logging agent) that runs one, and only one, pod per node. 
• TLS and L7 support (via the Beta Ingress API) provides a straightforward way to integrate into custom networking environments by supporting TLS for secure communication and L7 for http-based traffic routing. 
• Graceful Node Shutdown (aka Node Drain) takes care of transitioning pods off a node and allowing it to be shut down cleanly. 
• Custom Metrics for Autoscaling now supports custom metrics, allowing you to specify a set of signals to indicate autoscaling pods. 
• New GUI allows you to get started quickly and enables the same functionality found in the CLI for a more approachable and discoverable interface.

• And many more. For a complete list of updates, see the release notes on github. 

Community 

Documentation 

Roadmap 

Connect 

Methodology 

1. API responsiveness¹: 99% of all API calls return in less than 1s 
2. Pod startup time: 99% of pods and their containers (with pre-pulled images) start within 5s. 

API responsiveness for user-level abstractions² 

1. Create roughly 3xN ReplicationControllers of different sizes (5, 30 and 250 replicas), which altogether have 30xN replicas. We spread their creation over time (i.e. we don’t start all of them at once) and wait until all of them are running. 


2. Perform a few operations on every ReplicationController (scale it, list all its instances, etc.), spreading those over time, and measuring the latency of each operation. This is similar to what a real user might do in the course of normal cluster operation. 


3. Stop and delete all ReplicationControllers in the system. 

Pod startup end-to-end latency³ 

1. Create a single ReplicationController with 30xN replicas and wait until all of them are running. We are also running high-density tests, with 100xN replicas, but with fewer nodes in the cluster. 


2. Launch a series of single-pod ReplicationControllers - one every 200ms. For each, we measure “total end-to-end startup time” (defined below). 


3. Stop and delete all pods and replication controllers in the system. 

Metrics from Kubernetes 1.2 

API responsiveness 

Pod startup end-to-end latency 

How did we make these improvements? 

• Created a “read cache” at the API server level 
  (https://github.com/kubernetes/kubernetes/issues/15945 )
  
  Since most Kubernetes control logic operates on an ordered, consistent snapshot kept up-to-date by etcd watches (via the API server), a slight delay in that arrival of that data has no impact on the correct operation of the cluster. These independent controller loops, distributed by design for extensibility of the system, are happy to trade a bit of latency for an increase in overall throughput.
  
  In Kubernetes 1.2 we exploited this fact to improve performance and scalability by adding an API server read cache. With this change, the API server’s clients can read data from an in-memory cache in the API server instead of reading it from etcd. The cache is updated directly from etcd via watch in the background. Those clients that can tolerate latency in retrieving data (usually the lag of cache is on the order of tens of milliseconds) can be served entirely from cache, reducing the load on etcd and increasing the throughput of the server. This is a continuation of an optimization begun in v1.1, where we added support for serving watch directly from the API server instead of etcd: https://github.com/kubernetes/kubernetes/blob/master/docs/proposals/apiserver-watch.md. 

Thanks to contributions from Wojciech Tyczynski at Google and Clayton Coleman and Timothy St. Clair at Red Hat, we were able to join careful system design with the unique advantages of etcd to improve the scalability and performance of Kubernetes. 

• Introduce a “Pod Lifecycle Event Generator” (PLEG) in the Kubelet (https://github.com/kubernetes/kubernetes/blob/master/docs/proposals/pod-lifecycle-event-generator.md) 
  
  Kubernetes 1.2 also improved density from a pods-per-node perspective — for v1.2 we test and advertise up to 100 pods on a single node (vs 30 pods in the 1.1 release). This improvement was possible because of diligent work by the Kubernetes community through an implementation of the Pod Lifecycle Event Generator (PLEG).
  
  The Kubelet (the Kubernetes node agent) has a worker thread per pod which is responsible for managing the pod’s lifecycle. In earlier releases each worker would periodically poll the underlying container runtime (Docker) to detect state changes, and perform any necessary actions to ensure the node’s state matched the desired state (e.g. by starting and stopping containers). As pod density increased, concurrent polling from each worker would overwhelm the Docker runtime, leading to serious reliability and performance issues (including additional CPU utilization which was one of the limiting factors for scaling up).
  
  To address this problem we introduced a new Kubelet subcomponent — the PLEG — to centralize state change detection and generate lifecycle events for the workers. With concurrent polling eliminated, we were able to lower the steady-state CPU usage of Kubelet and the container runtime by 4x. This also allowed us to adopt a shorter polling period, so as to detect and react to changes more quickly. 








• Improved scheduler throughputKubernetes community members from CoreOS (Hongchao Deng and Xiang Li) helped to dive deep into the Kubernetes scheduler and dramatically improve throughput without sacrificing accuracy or flexibility. They cut total time to schedule 30,000 pods by nearly 1400%! You can read a great blog post on how they approached the problem here: https://coreos.com/blog/improving-kubernetes-scheduler-performance.html 


• A more efficient JSON parserGo’s standard library includes a flexible and easy-to-use JSON parser that can encode and decode any Go struct using the reflection API. But that flexibility comes with a cost — reflection allocates lots of small objects that have to be tracked and garbage collected by the runtime. Our profiling bore that out, showing that a large chunk of both client and server time was spent in serialization. Given that our types don’t change frequently, we suspected that a significant amount of reflection could be bypassed through code generation.
  
  After surveying the Go JSON landscape and conducting some initial tests, we found the ugorji codec library offered the most significant speedups - a 200% improvement in encoding and decoding JSON when using generated serializers, with a significant reduction in object allocations. After contributing fixes to the upstream library to deal with some of our complex structures, we switched Kubernetes and the go-etcd client library over. Along with some other important optimizations in the layers above and below JSON, we were able to slash the cost in CPU time of almost all API operations, especially reads. 

• Other notable changes led to significant wins, including: 
• Reducing the number of broken TCP connections, which were causing unnecessary new TLS sessions: https://github.com/kubernetes/kubernetes/issues/15664
• Improving the performance of ReplicationController in large clusters: https://github.com/kubernetes/kubernetes/issues/21672

In both cases, the problem was debugged and/or fixed by Kubernetes community members, including Andy Goldstein and Jordan Liggitt from Red Hat, and Liang Mingqiang from NetEase. 

Kubernetes 1.3 and Beyond 

Conclusion