--- Hello! Imagine a thing with human faces, what a treat, I get to stand up, not worry about being on mute, use my clicker and everything! --- So, to kick things off my name is Chris Nesbitt-Smith, I'm based in London and currently work with some well known brands like learnk8s, control plane, esynergy and various bits of UK Government I'm also a tinkerer of open source stuff. I've using and abusing Kubernetes in production since it was 0.4, believe me when I say its been a journey! I've definitely got the scars to show for it. We should hopefully have time for questions and heckles at the end, if not come find me afterwards. --- The history of the pets vs cattle terminology is muddy, most link to a presentation Bill Baker from Microsoft made in 2006 around scaling SQL server. --- Way back then in the before times, we called ourselves sysadmins and treated our servers like pets --- For example Bob the mail server. If Bob goes down, it’s all hands to the pumps. The CEO can’t get his email and it’s the end of the world. We do some incantations, make some sacrifices at an alter and resuscitate Bob bringing him back from the dead --- Crisis averted, cue the applause and accolades for our valiant sysadmins who stayed up late into the night --- In the new world however, servers are numbered or maybe uuids, like cattle in a herd. --- For example, www001 to www100. When one server goes down, it’s taken out back, shot, and replaced on the line. --- Why am I telling you this rather morbid story? Kubernetes deals with that right? and saves us from the tyranny --- And you're right, it does. All you're computers are called nodes and abstracted and given arbitrary names, autoscaling groups and such will automatically detect the sick in your flock, take them out, and bring a replacement in. all while seamlessly (ish) rescheduling the workload that was on the failed computer --- And Kubernetes takes that a step further, your workload also has unique names --- Like the physical servers your workload failures can be detected, and replaced seamlessly --- So wheres the pet? --- well.. --- Whats the first thing we do with a brand new Kubernetes cluster? --- Hint: it's not deploying your application or anything the business cares about --- Look familiar? yeah, we had to do a load of 'things' just to make this cluster able to start running our workloads --- And it's worth noting that with a trend towards more and more features being being 'out of tree' that is to say they're optional add-ons and don't ship with core Kubernetes. Examples of this are things like flex volumes, policy and basically all the Kubernetes sig projects that many find essential is only exasperating this issue --- That might work for when you've got a single cluster But what about when you've got dev integration staging qa that your app needs to run on --- Or worse, when you need separation between your teams or products --- Maybe you've automated that, bash, ansible, terraform, whatever you like, cool good on you --- However you'll find it won't be long before theres an updated version perhaps patching a vulnerability you care about and you may be stuck trying to test every single app across your estate --- This is what we're calling day 2 operations, we used to call it BAU or business as usual, and it's where reality catches up with our idealistic good intentions --- You'll quickly find that clusters are running various versions, given the rate of change in the community its unrealistic to run :latest everywhere confidently without breaking production and disrupting your operational teams. --- Permutations of seemingly common tool choices, some teams might use kong, others nginx, another apache, all for good reasons I'm sure --- Seemingly infinite possibilities across the estate emerge --- Sad times --- Congratulations, you're now the proud owner of a pet shop, or if you managed to automate the creation --- You can call it a pet factory, but it's a headache --- So what, how does this hurt you might ask? --- Maybe you like pets? --- Well, presuming of course you're in cloud, your world could roughly be summarized into tiers Apps, well these are things that your board room know about, and can probably name, so think your public website, shopping cart system, customer service apps, online chat interfaces, email system etc. These are all implicitly providing some value in of themselves to your end customers. --- Infrastructure, with cloud this is all commodity thankfully, the days where anyone in your business caring about the challenges of physically racking up hardware, not overloading the weight in the cabinet, taking pride in how well they've routed cables have hopefully passed; and you're consuming infrastructure, hopefully you've codified this but even if you're in to ClickOps, making sure its running is not your problem. No one in your business is concerned with hardware failures, patching routers every-time theres a critical vulnerability, testing the UPS and the generators regularly, upgrading the HVAC when you add more servers. --- "YAWN-orarma" as my 16 year old would say and curse me for repeating. Your interactions with any of this is a few clicks or lines of code and some infra is available to you with an SLA attached to it. --- If only the story ended there --- But sandwiched between those is a grey layer, of all the operational enablers, its where your 'devops' or 'SRE' team live. So think log aggregation, certificate issuers, security policies, monitoring, service mesh and others. These are things you do because of all sorts of reasons ranging from risk mitigation to emotion and technically unqualified opinion or just without foresight of what was round the corner in 6 months. --- Let's just make the leap and assume for a minute you are more technically competent than your goliath multi-billion dollar cloud vendor --- You've completely negated many of the benefits of going to cloud in the first place by ripping up the shared responsibility model All of this while technically fascinating for people like me to stand and --- stroke my beard at. This is delivering absolutely zero business value, unless of course your business is building or training on those products. --- and who'd want to get into that business! --- And thats not all! Recruitment... --- You might think you want a devops right? oh no wait, devops with Kubernetes experience, maybe a CKA? oh yeah, its on AWS, and we use linkerd and in some places istio, no not the current version, or even the same version everywhere. a mix of pod security policy, kyverno and OPA for policy, some terraform, helm, jenkins, github action soup going on, all in a mono-repo apart from all that stuff that isn't. --- We're well outside the remit of commodity skills and back to hunting unicorns. --- Sure you'll find some victims. sorry... --- I mean candidates; that you'll hire, well now you've got one hell of an onboarding issue before they can do anything useful and help your business move forwards faster than it did without them. --- And if you hired smart people they'll come with experience and their own opinions of what worked for them before, so your landscape gets bigger and bigger and more complex and diverse --- I did some googling, this is what the CNCF landscape looked way back in 2017. --- Choices, right? choices and logos as far as the eye can see. --- Have you seen it recently? --- This has got a bit out of hand, I'd say someone aught to have a word but I suspect that'd just make things worse by adding yet another thing --- and don't get me started on operators, nice idea but betray any ideals of immutability, crazy levels of abstraction for.. --- and have you seen the crazy of mutating admission controllers --- If you're really mad, you can nest these things, with operators that create crds for other operators that are all mutated, heaven forbid someone bumps the version of anything? --- All no doubt held together with sticky tape, chewing gum, glue, pipe cleaners, thoughts and prayers and --- helm --- a string based templating engine where any community module has to eventually expose every parameter in every object file abstracted by a glorified string replace --- So now I've got to have in my head all the complexities of a linux/windows host, how the container runtime works, the software defined network and storage, the hypervisor, before the container, the scheduler, controllers, auth and policy and mutating policy in the cluster. --- before I worry about how someone in the nested helm chart mess of hell, has mapped the replica count of one of the deployments to a string called db replica count, and how that has changed in a new version of a dependency not following semver to "database_replica_count", so instead of having my expected 3 I've now only got 1 --- when I could have just written a yaml patch for the replica count in the deployment object of the database resource using stable API versioning with schema validation for free, ahhh --- the kids doing Kubernetes don't seem to have learned from the past --- don't get me wrong, I love the open source community with all my heart, its so important --- and its simply not possible to do anything without it sorry, not sorry, yes as a sidebar, every talk this year is contractually required to reference log4j, this is my slide, deal with, its not relevant, it can come out in a couple of months --- everything literally everything that exists around us depends upon it, and the community is brilliant, at building some truly remarkable very high quality things, but we must accept that --- the open source community --- is awful at packaging things up --- in this way for consumption, introducing needless abstractions --- but enough of that, I'm definitely going to hell now --- happy place chris, happy place where was I, right yes so through all of this --- I can't possibly think of a faster way to go from enthusiastic engineers playing with new exciting tech --- To deeply unhappy ones trying to fix something at 4am --- and before they can do anything meaningful they've got an orienteering exercise to switch mental context to whatever the intended permutation of things it is they're looking at. --- Meanwhile your business value delivering apps are offline, or worse at breach --- Rewind a minute we didn't want any of these things, how did we get here? What can we do about it? --- honestly? bin it all --- kill it with fire --- and then Learn to --- love vanilla, vanilla is great, and delicious too --- anyone remember KISS? --- no, not the band --- Keep it stupid simple --- or Keep it simple, stupid --- and embrace the shared responsibility model on offer, and make your cloud vendors do more than just provide compute, turns out as it happens, they're not that bad at it --- I'm not daft I know it's not sexy and exciting, you might even find recruitment harder if you're used to --- hunting magpies who follow the shiny and don't like boring stuff that works --- So, to answer the question posed from the title of my talk, is it time you put your pet Kubernetes cluster down? --- Yes, yes is it. And in the immortal words of s-club 7 if you can --- bring it on back immutably from code, all without anyone noticing (I'm referring to the original version of the lyrics) --- Then maybe just maybe it can earn the right to stay to --- die another day --- I've been Chris Nesbitt-Smith, thank you again for joining me today and enduring my self loathing. Like subscribe whatever the kids do these days on LinkedIn, Github whatever and you can be assured there'll be no spam or much content at all since I'm awful at self promotion especially on social media. cns.me just points at my LinkedIn. talks.cns.me contains this and other talks, they're all open source. --- Questions are very welcome on this or anything else, I'll hold the stage as long as I'm allowed, or find me afterwards, this grumpy old man needs to go find somewhere to sit down soon