Moving stuff between Kubernetes clusters can be a pain in the butt. You have to do this when:
Migrating between container platforms, such as EKS -> AKS
Upgrading clusters when you want to upgrade in parallel (move everything from old to new) as opposed to upgrading in-place (update one cluster from old to new). This would be something like what we’ll talk about in a minute, going from OpenShift 3.x to OpenShift 4.x
Moving work/applications between clusters, e.g. two clusters in different datacenters
Migrating work between clusters requires some thought and planning, and good solid processes. Specifically, the migration from OpenShift 3.x to OpenShift 4.x requires a parallel upgrade, because there is no in-place upgrade available for all of the new goodies in RHEL CoreOS (the underlying infrastructure of the cluster). OpenShift 4.2 released recently, so we thought it would be good timing to put our migration thoughts down here. However, the advice below is generally good for any Kubernetes cluster parallel upgrade or other migration.
One of the most crucial metrics of success for an enterprise application platform is if the platform can protect: a) the applications running on it, and b) itself (and its underlying infrastructure). All threats to an application platform eventually come from something within that platform – an application can be hacked, and then it attacks other applications; or there could be a privilege escalation attack going after the underlying host infrastructure; or an application can accidentally hoard platform resources, choking out other apps from being able to run.
We work with a lot of people who are implementing Continuous Delivery. We see that when various bumps and boulders get out of the way of delivering software stably and rapidly, there’s a strong push to go very very fast. When this happens, there are often barricades put up in the name of security – because traditionally speed and security have been considered enemies. Traditional enterprise IT security would say, you can’t possibly go fast in a safe way,
There is a dream that lives in IT – it is the dream of the easy button. Push one button (or even a couple of buttons, we’re flexible!) and get immediate value. Everyone wants these easy buttons, and every software sales company wants to sell these easy buttons.
A long time ago (comparatively, time moves very oddly these days…), I (Laine) wrote this post about my experience thus far with the first church I’d ever regularly attended. Coming to Christianity as an adult has been…an experience. Especially as a strong, capable, independent, female (I wish it didn’t matter, but I’m not convinced it doesn’t) adult who is as committed to God as I know how to be.
See, the thing is, I came to Christianity as an adult – I did not come to faithas an adult. My faith is independent of any church, and my relationship with God is the oldest, strongest relationship I have.
We all have good parts of our lives: fun events, good friends, adventures, trying new experiences. We all have bad parts too – deaths of people we love, arguments that end relationships, work disappointments.
In my life, I have had some amazing experiences and some really low lows.
Recently, however, I’ve noticed that something seemed to be broken with how I experienced the good in my life. Even really great things, I didn’t enjoy. I didn’t really notice until I could experience them normally again, but it was like I went numb. I would notice that I couldn’t taste my favorite food, or a delicious cigar…and I would wonder, what is going on?
I realized, some good things are so Big Good that it’s actually hard for me to process them. I get scared…and then I hide from the good. This is super annoying actually, and double bad, because it prevents me from both enjoying the good and also being thankful for the good.
We’ve talked previously about how developers drive organizational success: they deliver the applications by which companies deliver their competitive advantages. Because they are a way for companies to deliver products to customers, those delivered applications are critically valuable. Application development is a lot like extracting gold – it creates value out of raw resources.
Application development is a lot like extracting gold – it creates value out of raw resources.
Gold, wealth, needs to have some amount of protection.
We both recently started traveling a lot for work. Luckily we both like to travel, or it would be kind of terrible – because traveling is a lot of work. We’ve learned a lot about travel in general, and especially work travel, mostly via stumbling across pro tips.
Also…we really think that if you do a thing a lot, you should try to get good at it. You’ll have a lot more room to enjoy the fun parts that way.
If you do a thing a lot, get good at it. You’ll have a lot more room to enjoy the fun parts that way.
We kept saying we should write up a blog about travel tips, so hey we finally did it! Here are our favorite tips, the ones that have helped us the most – we share this with the hope that it helps you. Most of these are focused on work travel, although a lot of it will work for fun travel too. If you know tips we don’t, please drop us some wisdom in the comments, because honestly we’re still learning a lot…
For All Travel Methods
Join Some Loyalty Programs
Pick an airline, and a hotel chain, and use their loyalty programs. Points add up fast! We both like Delta and Marriott, but Laine travels to more places and also has accounts with United and Hilton (we don’t get anything if you click on those links, Laine checked…).
Match Up Your Credit Card Rewards
If you’ll be charging your travel expenses and being reimbursed, it can help to have those expenses on a different credit card than your normal expenses. If you go this route, pick a credit card with good rewards, because chances are good you’ll get to keep those (see above, re: airline and hotel rewards programs!).
Buy Some Grown Up Luggage
You’ll need good, well-made luggage. Luggage is one of those things where you really do get what you pay for. Laine likes Samsonite (which is often on sale at Kohl’s, but Google as you will) soft shell (and therefore expandable) cases in as bright of colors as humanly possible, Josh has a hard-shell case that looks like it could probably withstand an apocalypse or two. Get two different sizes – one that you’ll check when you fly/for longer trips, and one that will fit in the overhead bin and can be a carry-on (more on that in a minute…).
Get a Frickin’ Bluetooth Headset Thing Already
(Amazing Josh rant ahead!)
This is a pet peeve of mine. If I have to walk by another speakerphone conversation while I’m walking, and listen to them yell at their phone and then try to hear over the ambient noise…
…or even worse, holding their phone in their hand and trying to talk on speaker while they’re driving…
…I’m gonna yell at them, and their phone.
(…end, Josh rant.)
It’s much, much easier on your brain to have a headset and talk hands-free. It lets you think less about your hands and more on what you’re supposed to be working on. Most important, it’s kinder on the person you’re talking to, because they don’t get echos of themselves plus all the background noise. A good headset will automatically filter out background and wind noise, so it’ll always sound like you’re sitting in an office, instead of driving or walking outside.
We love the Plantronics Voyager Legend, which, again, we get bupkis if you click on that link (unless you’re from Plantronics/Poly, in which case will work for extra headsets. :D)
Make Duplicates
Packing and re-packing means you’re probably going to forget stuff. One way to avoid this is to have duplicates of the things you can – the coolest example of this was Josh’s coworker who recommended multiple full cosmetic bags. Like…buy an extra toothbrush/toothpaste/face stuff/etc, and keep it in a cosmetic bag that just lives in your suitcase. We extended this pro tip to two cosmetic bags – one for short trips/the carry-on, because you’ll be limited to one clear quart-size ziploc bag of liquids that are each less than 3oz, and one for the big/checked suitcase that has all the things you might need. Think about stuff like nail clippers and tweezers here too!
Sometimes hotel shampoo/conditioner/soap/lotion is good enough, and sometimes it’s not.
Josh: I depend on hotel stuff and keep a little bit of liquor in my quart bag.
Laine: I use hotel shampoo and conditioner, but have stupid sensitive skin and prefer not to chance it with things like lotion/face stuff.
Other things you can and should duplicate between luggage:
medication – luggage can get lost, and you will NOT want to mess around with trying to get an emergency refill in a new city – especially if you take any kind of controlled substance
cords/plugs/batteries – you’ll need to charge your phone, laptop, headset, Kindle, etc…USB hubs or combination cables are great for this
Pack Extra Clothes
Josh: If I check a bag, I always pack one extra pair of clothes: jacket, shoes, pants, socks, shirt, underpants. Just in case.
Laine: I always pack an extra of everything, because I am messy and I tend to spill stuff on myself. I typically pack multiple extra, especially of shirts, because my sensory issues are such that I never know exactly what my brain will refuse to wear on a given day. Being comfortable with traveling is super important.
There are Probably Stores Where You’re Going
Did you know Walgreen’s has great socks? True story.
Don’t worry too much about packing exactly the right thing. That can make travel stressful and stupid.
There are stores for travelers for almost everything, and you’ll figure it out. Things obviously cost money in stores, and that’s a valid concern – but just like anything else, there’s a threshold of money vs worry and if the worry is too high of a cost, assume that money will help.
Just don’t forget the same thing multiple times or you’ll end up with like six USB-C cables by your luggage at home…
How are You Going to Get There? (drive vs fly and the 4-ish hour rule)
Of the time you spend away from home (away from Netflix or Reddit or blogging or playing Unturned with your kid(s)…), the time actually moving from place to place is the biggest potential waste. So…plan how to minimize it.
Josh: Here’s my thinking: if the total drive time from door to door is less than four hours, I should just drive it. Even with TSA Pre-Check (see below), driving to an airport that has good direct flight options (…see more below) is a 90 minute drive. Add in parking, how much I hate security checkpoints, needing to be early for the flight, the stress of maybe missing flights, limited airport meal choices, and sitting close to 100 other people…then after landing, getting my bags, finding transportation, and being limited to Uber/Lyft + my own two feet if I don’t want to bother with a rental… I should just drive it. It’ll be faster. And less annoying.
Plus flights are expensive and driving is often cheaper – and if you get reimbursed for mileage, you get paid rather than the airports.
Laine: I hate driving, especially alone. ADHD things. It’s boring, and my mind wanders, and I have to sit still the entire time. My phone giving me directions is better than the days of printed-out MapQuest, but… ughhh. Josh is right that the question really starts around 3.5-4 hours, though – I recently went to the same airport two weeks in a row. The first time, it took 4.5 hours to fly when it would have taken 3.5 to drive. The second time, it took 3 hours to fly when it would have taken 4 to drive. This science was also with a very close airport and a direct flight – anything else, driving probably is better.
…also, that first flight was delayed for 4.5 hours, so I guess it really took 9 hours to fly it? Time math is hard.
Flying
TSA Pre-Check
If you fly, get TSA Pre-Check or Global Entry or CLEAR. This post does a great job at explaining the difference, but the basic idea is that these programs allow you to trade money and some personal information for a) faster security lines, and b) not needing to do all of the complicated security things. Pre-Check (domestic) and Global Entry (international, includes Pre-Check benefits) are run by the TSA itself, and CLEAR is run by a private company backed by Homeland Security. CLEAR can also be used at some stadiums/large venues.
Laine was sold on Pre-Check when she realized she didn’t have to take off her shoes or wrestle electronics out of carry-on luggage. We mentioned it was work travel, right? Do you know how many electronics nerds carry for work travel? More than zero. It was only after we tried it a few times that we realized how much faster it is to get through security.
TSA Pre-check is quick to get (we had our known traveler numbers [KTN’s] within 3 business days of the in-person part), costs $85, and lasts 5 years. So worth it for skipping lines if you fly more than once or twice a year.
Pre-Check also lets you bring children under 12 with you through the fast lane, although the other two require every traveler to register.
Fly Direct
Don’t bother with transfers. Transfers are stupid and annoying and risky. Fly direct. It’s faster, it’s often cheaper, and it’s always easier on your tired brain not to need to worry about missing a connecting flight, or to bolt through an airport at the last minute.
To Check a Bag or NOT to Check a Bag – That is the Question
If you’re flying, think about if you need to check a bag or not. Most airlines allow you to have one “carry-on” and one “personal item” – which can be like…a backpack, or a large purse – without needing to check a bag.
Reasons to check a bag:
You can bring more cool stuff, like alcohol, guitars, and spare shoes.
More room for spares for stuff. Or loot.
You can pack your full-size cosmetic stuff/liquids.
Reasons to not check a bag:
Most airlines charge a fee per checked bag.
It takes more time before and after your flight.
You gotta lug more stuff around.
Josh: My rule is 2 days. I can pack for 2 overnights in a carry-on bag, and I don’t keep a big one.
Laine: I can do 3 days, but I really hate the time suck that is checking a bag and would rather pack lighter.
Driving
Get an EZPass or equivalent
If you’ll be driving through states that have toll roads, it turns out that stopping to pay for tolls is dumb. Take a second and get yourself whatever the variant is of a pre-paid pass for the state(s) you’ll be driving in. You can find a list here. We know the most about the E-Z Pass, which works in most of the states in the Midwest.
Don’t Try to Drive + Work for More than About 10 Hours/Day
Josh: If I have meetings with my team(s) and/or my customers, and a drive time that adds up to more than ten hours, I get exhausted. Get a hotel and rest. Not crashing in a car wreck is more important than getting somewhere rapidly.
Stop. Pee. Get Coffee. Be Early!
On the way to somewhere, take breaks, at about 3 hours intervals or whatever works best for your brain.
Leave early so you have time to take these breaks and still arrive early – both because spending four hours worried that you’re going to be late is a lot of stress, and also so that you can do what you need to do to wind down before you have to actually work. Leave yourself enough time for a walk, or a smoke break, or to call someone who helps your brain.
Spending four hours worried that you’re going to be late is a lot of stress. Leave early so you’ll be early.
Conclusions
Like we always say: take care of your brain. You need it to do your job well, and to live your life well. Traveling is a blast, but it’s also definitely work – learn what makes it easier on you, and plan for that. Some of our tips will probably sound ridiculous, and that’s totally fine – the point is actually more that it’s really important to pay attention to how you function best.
These tips, however, are what we’ve found work best for us! Again, if we missed anything, please let us know!
There have been many words written about the Capital One breach – but a lot of them didn’t explain what actually happened. We care a lot about security in general, and cloud security in specific, so Josh set out to find some words that did explain what happened:
The Krebs article might be the best for this. However, as far as we could tell, no one’s tackled it from a “what can enterprises learn from this?” standpoint, and…that’s what we really care about.
TL;DR: The Event
A hacker named Erratic, who was a former AWS employee, took the following actions:
Owned the Web Application Firewall (WAF) on Capital One’s Amazon Web Services (AWS) Virtual Private Cluster (VPC)
Used the credentials of the WAF to connect to other AWS resources in the VPC, including their storage objects (S3 Object Stores)
Synced (copied) the object store to her own storage
“With this one trick, you can get 100M Credit Card Numbers! The secret THEY don’t want you to know!”
– Best ClickBait Ad Ever
So…there’s a lot about the mechanics of this that’s unclear. But we can explain what seems to be widely accepted as fact. First, some definitions:
A Web Application Firewall (WAF) is basically an entry point into a system – it isn’t intended to be entry, though, it’s intended to be a layer of defense.
AWS is Amazon’s public cloud.
A virtual private cluster (VPC) is a cordoned-off part of a cloud – so, it was an area of AWS that was specifically for Capital One.
So…
Somehow the hacker Erratic was able to log in to one of Capital One’s WAF.
From there, she got to their storage objects that represented information about people – specifically, people who had used the business credit card application…application. Overloaded words are the best!
Finally, she copied those storage objects that represented people to her own area of AWS – like copying a file from someone else’s Google Drive into your Google Drive.
Questions Outstanding
…there are a lot.
It’s not clear to how Erratic did #1, logging in to the WAF. The most likely answer is that the username/password was something not complicated enough – like maybe the default of admin/admin. But there are also other possibilities, and if Capital One has disclosed this piece, we couldn’t find it.
There are a few ways step #2 could have happened – the WAF could have already had access to all of the storage objects, or Erratic could have given the WAF direct access to the storage objects. The J Cole Morrison article above explained one possible scenario: Amazon IAM could have been used to take advantage of the fact that she was already in the WAF and then extended the default trust of “well, you’re in the WAF, so okay” – security people call this a “pivot”.
Step #3 is basically…copy/paste. There are probably some interesting nuances here, like…if she didn’t give the WAF authority to read the objects, why did the WAF have the authority? What business use case would require giving an access point read access to an entire store of customer data? Also she would have had to have given something access to write to her own AWS space, at least temporarily.
The Pain: $100M-$150M
The Capital One press release stated that this incident will “generate incremental costs of approximately $100 to $150 million in 2019.” Capital One was one of the earliest companies to go to AWS/the cloud, and they made a lot of noise about it – here, and here. Explaining technology success is one of our favorite things, but there are trade offs if you could otherwise manage to keep your backing infrastructure a secret.
This has lead to egg on AWS’s and Capital One’s faces, which is unfortunate, because this really doesn’t have much to specifically do with AWS or clouds in general….
…or does it?
– Not Intended to be ClickBait
Clouds in General
This isn’t the first AWS data breach (see end of the blog for a list of others). The list is not small, unfortunately.
Please raise your hand if you are sure you haven’t been hacked?
We’re gonna say this is partially because AWS is the biggest, been around the longest, and had to figure out hyperscale stuffs without anyone to copy from because they were the first.
But still… yikes.
A big part of this is that Amazon makes things super easy. So easy a caveman could do it, right? And…that’s the trick. It’s super easy to type in a credit card (or even an AWS gift card, I (Josh) have one they gave out at a trade show) and spin up some storage and compute. Unfortunately, it isn’t super easy to spin up security tailored to clouds.
We used to have to wait for infrastructure teams in our data centers to (hopefully) handle the security for us. They’d put your request in a queue, and get to it a week later…then they’d ask the storage admins and VM admins for some data and some compute, and that request would go into a queue…and then, several steps later, the firewall admins would get involved…but doggone it, eventually the pros would secure things like they were trained.
VM-based infrastructure has been around a long time, and the kinks have been worked out. Cloud infrastructure is newer, and exponentially faster to use – that’s one of the biggest appeals. Unfortunately, because it’s newer and because it’s so fast, kinks still exist – and one of the biggest is how to make it secure without slowing down the people using it.
Clouds are not all insecure, the sky is not falling – but they do require more deliberate attention to security than perhaps we’re used to in most of IT.
Takeaways and Recommendations
With Infrastructure as a Service that’s as fast and easy as cloud-based, it’s clear that there are often times when the right security-aware folks are not involved. It’s extremely easy to get going with platforms like these, which is…kind of the point. Simply put, you can get insecure systems faster and easier than you can get secure systems – for now, anyway. The industry knows this, and is trying to make it better.
Until security catches up to the speed of IaaS, companies need people who can secure their systems to be involved in setting up new platforms, and setting up best practices for their use. The balance point of that is not removing too much of the speed and agility gains of advances like IaaS because of security – ideally security should be something that everyone agrees is worth the trade.
So…after all of that, here are some recommendations:
Single layers of security are not enough. You need Defense in Depth, and vital areas like customer data need to be strongly protected regardless of the platform trying to access them.
Security practices and implementations should be transparent, at least within a company, and questions should be welcomed and encouraged. Open culture helps with security, too.
Security should be automated as much as possible, and that automation should also be transparent (infrastructure as code).
Enterprises need to choose platforms that are secure, that have people dedicated to the security of that platform as their job.
We really love this blog. We started it almost exactly 6 months ago and it means a ton to both of us. We started with two posts a week – and then Josh started a new job. We downshifted to one post a week – and then Laine got a new job. We’ve managed to keep on keepin’ on at one post a week since then, which… well, we really love this blog.
One of the first things we ever did that made us sit up and realize that maybe we made a seriously effective team was give a nerd presentation – we talked about feature toggles as an architectural concept. A few months after that, we went to UberConf in Denver. That was Laine’s first IT conference, and we had a blast. That’s a pretty good “God does stuff everywhere” story, which we should probably tell at some point…
After that conference, as we adjusted back to normal life, we talked about how seriously amazingly cool it would be to give nerd presentations at a nerd conference of that level – national, and with nerd-famous people like Mark Richards and Neal Ford. Josh definitely fanboy’d when Mark Richards included him in a demo in a presentation. We also befriended one of the speakers on the tour, who lives nowhere near us. We filed away the plan to some day speak at national nerd conferences in general, and at UberConf specifically, in the “haha, sure, that might happen some day” file.
We called this a goal, but…it was a dream. It was a dream in the way that little kids gleefully dream about being an astronaut when they grow up.
Laine was off work for 6 months. Again, another story for another time. But while she was off work, we started to apply to speak at conferences. Josh’s new job was friendly to the idea, Laine had no job, it was something to think about, so…we sort of figured why not.
We applied to speak at O’Reilly’s Open Source & Software Convention (OSCON), who was having a themed Inner Source day this year. Once Laine understood what on Earth “inner source” meant, we were sort of like, “hey it is us and one of the things we love the most!!” We submitted two talks.
We also started conversations about getting onto a No Fluff Just Stuff stop, semi-local – NFJS organizes UberConf along with a lot of other regional conferences, all throughout the year. The other major conference they organize is ArchConf, in December – which was also on our Nerd/Astronaut Dreams Bucket List.
And then, on a Friday afternoon, we found out the following:
One of our talks was accepted for OSCON.
One of the speakers for UberConf had to drop out, there were some spots open, and we could have them if we wanted.
God does weird, wonderful, lavish, unexpectedly awesome stuff.
…you mentioned a hiatus?
Yes! We did.
OSCON and UberConf are the same week, the week of July 15th. We got lucky (jklol pretty sure it was God doing more awesome stuff) and our talk at OSCON is that Tuesday, and our talks (4!!)at UberConf are Wednesday and Friday. So…we decided to do both conferences.
J: Should we do both?
L: Are we really crazy enough to try that? :thinking:
Us: Yep!!
We’re getting ready for those talks now. We are both extremely dedicated, prolific workers, but even we have limits. We have several posts in varying stages of done, but the kinds of thing we write require focus and attention and time and soul – and we pretty much only know how to make any content we produce in that same way.
“A man’s got to know his limitations.” – Harry Callahan, Magnum Force
We will be back. We have so many thoughts and feels and did we mention we love this blog?
Logistics
These are the descriptions and scheduling of our talks:
Please come say hello if you’ll be at either OSCON or UberConf. (If you are not attending and would like to, we have discount codes!) We love these topics, we love talking about them, and we are so stupid excited to be doing this.
