Search for a command to run...
Don't bother!
I'm a jack of all trades with deep expertise in infrastructure, cloud, networking, and devsecops. In my spare time I also like to play around with self-hosting, embedded devices, camping, permaculture, sailing, and scouting
No comments yet. Be the first to comment.
With Typescript and CDK
Lets think about why we allow streaming updates to production, and what might be some other options
DampRid is great stuff for dealing with excess humidity. Lots of boaters (and hunters, and countless other folks) use it to keep less frequently used spaces fresh and free of mold and mildew. The only real downside is that it's expensive. A 3 pack of...
Overall, the boat is in pretty good shape, or so it seems to me. Reminder: I'm very new to boat ownership - This is my first! After giving it a thorough once over, I've worked out the following... Airflow Unfortunately, the docks at the lake do not h...
There are many ways of handling your AWS Root credentials, but after many years of going back and forth with various vaults and password management systems, I came across a surprisingly simple pattern: Don’t bother remembering those passwords to begin with! Set up a 2fa token, and just use the password reset workflow.
We’ve all been there… We set up a new AWS account and new we have an MFA token and password to deal with. Do we use a physical security token (single point of failure)? If so where do we store it so that it’s accessible? If you have a 24×7 staffed NOC with a vault, this is a simple solve, but most places don’t have that. Especially in the SMB and startup spaces. So we go to a virtual MFA device. And now only one person has that on their phone (single point of failure again) or we need to share the key out (hacky) or store in in a vault of some kind (at least these are digital).
Well, here is one approach that I’ve actually used several times with surprising success. It has also withstood many audits. Does that mean it’s perfect? No. But it is an effective method, even it it’s a bit out there.
An email provider that supports distribution lists. Ideally one that lets you manage those memberships dynamically based on security groups. I use Azure Entre (Active Directory) and Exchange Online (all via Office365) for this.
A place to securely store OTP keys, ideally tied to the same security group management tool as the email distribution lists. I like to use Bitwarden for this, but Hashicorp Vault and other tools support this as well.
For every AWS account you create, you will use the DL as the email address for the account. Your root account might be DL-AWS-Root@example.com while your shared dev account might be DL-AWS-Dev@example.com. Naming things is hard, but defining this namespace early and trying to stick with it is important to your long-term sanity as accounts sprawl. What do you use for the password? The longest string of random characters that your password generator of choice will spit out, and that AWS will accept. I believe that that is currently 64 characters. Copy it to your clipboard, use it to log in, then overwrite your clipboard. You never want to remember it. Just get rid of it. It never existed. Poof! It’s gone forever into the ether…
Now bootstrap your account as normal, add your support plan if you want one, setting up any core roles that you need to assume that aren’t part of your account factory, and finally, set up your MFA token for the root account. Store that token in your vault, and control access to it. Now that the bootstrapping is complete, log out. And make sure that root password is gone, never to be a thorn in your side again. Beautiful isn’t it?
Only certain people should ever use the root account. Generally, it’s a break glass kind of thing, and you want everyone to know it’s happening. Thankfully, you have an email distribution list set up for your password reset process. So any time someone tries to reset the password to log in, everyone on the list will know about the attempt. You also still have your second factor in a separate system. Then the individual logs in, and resets the password to another completely random string of 64 characters and purposely doesn’t remember it.
Though this approach is a bit out there, it does work, and it stands up to many security policies and controls frameworks. It’s still not perfect though.
It relies on the people in the process to not save the new password. This can be mitigated by creating alerts for whenever the root account is accessed, which is already a best practice.
It creates reliance on other systems, email and a vault, which have other administrators and attack surfaces. Granted, we have reliance on those systems anyway, but it’s still an extra hop. In a break glass scenario, we want fast access. We don’t want to wait for an email to go through.
Though the OTP key is stored in a separate system, access to it is typically controlled by the same authentication and authorization framework. This doesn’t have to be the case, but it does dramatically reduce operational burden.
It’s very counter-intuitive. We want to store important credentials, not forget them! As a result, this can take some time to work through the approvals process.
Now that AWS supports Passkeys, you could also store those in a vault instead, though there may be issues with that approach that I am not aware of. This approach might be worth diving into, though. Especially since Bitwarden and the like supports passkey storage as well.
Well, there it is. A surprisingly scaleable pattern for handling root credentials and tokens in a simple or complex AWS environment. What do you think? Is this approach out there? What other issues do you see with it? How do you manage your root credentials, and what issues do you have with your approach?
