README.md 11.6 KB
Newer Older
Afraz Ahmadzadeh's avatar
Afraz Ahmadzadeh committed
1
# Akinaka
2

Afraz Ahmadzadeh's avatar
Afraz Ahmadzadeh committed
3 4
This is a general all-purpose tool for managing things in AWS that Terraform is not responsible for -- you can think of it as an extension to the `aws` CLI.

5
At the moment it only does three things; blue/green deploys for plugging into Gitlab, AMI cleanups, and RDS copies to other accounts.
Afraz Ahmadzadeh's avatar
Afraz Ahmadzadeh committed
6

Afraz Ahmadzadeh's avatar
Afraz Ahmadzadeh committed
7 8 9 10 11 12 13 14
- [Akinaka](#akinaka)
  - [Installation](#installation)
  - [Requirements and Presumptions](#requirements-and-presumptions)
  - [A Note on Role Assumption](#a-note-on-role-assumption)
  - [Deploys](#deploys)
  - [Cleanups](#cleanups)
    - [AMIs](#amis)
    - [EBS Volumes](#ebs-volumes)
15
    - [RDS Snapshots](#rds-snapshots)
Afraz Ahmadzadeh's avatar
Afraz Ahmadzadeh committed
16 17 18 19 20
  - [RDS](#rds)
    - [Copy](#copy)
  - [Container](#container)
  - [Billing](#billing)
  - [Contributing](#contributing)
Afraz Ahmadzadeh's avatar
Afraz Ahmadzadeh committed
21 22 23

## Installation

Afraz Ahmadzadeh's avatar
Afraz Ahmadzadeh committed
24
    pip3 install akinaka
Afraz Ahmadzadeh's avatar
Afraz Ahmadzadeh committed
25

26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114
## Requirements and Presumptions

Format of ASG names: "whatever-you-like*-blue/green*" — the part in bold is necessary, i.e. you must have two ASGs, one ending with "-blue" and one ending with "-green".

The following permissions are necessary for the IAM role / user that will be running Akinaka:

    sts:AssumeRole

The following permissions are necessary for the IAM role that the above role /user will be assuming, if you wish to use every single feature:

    {
        "Version": "2012-10-17",
        "Statement": [
            {
                "Sid": "2018121701",
                "Effect": "Allow",
                "Action": [
                    "ec2:AuthorizeSecurityGroupIngress",
                    "ec2:DescribeInstances",
                    "ec2:CreateKeyPair",
                    "ec2:CreateImage",
                    "ec2:CopyImage",
                    "ec2:DescribeSnapshots",
                    "elasticloadbalancing:DescribeLoadBalancers",
                    "ec2:DeleteVolume",
                    "ec2:ModifySnapshotAttribute",
                    "autoscaling:DescribeAutoScalingGroups",
                    "ec2:DescribeVolumes",
                    "ec2:DetachVolume",
                    "ec2:DescribeLaunchTemplates",
                    "ec2:CreateTags",
                    "ec2:RegisterImage",
                    "autoscaling:DetachLoadBalancerTargetGroups",
                    "ec2:RunInstances",
                    "ec2:StopInstances",
                    "ec2:CreateVolume",
                    "autoscaling:AttachLoadBalancerTargetGroups",
                    "elasticloadbalancing:DescribeLoadBalancerAttributes",
                    "ec2:GetPasswordData",
                    "elasticloadbalancing:DescribeTargetGroupAttributes",
                    "elasticloadbalancing:DescribeAccountLimits",
                    "ec2:DescribeImageAttribute",
                    "elasticloadbalancing:DescribeRules",
                    "ec2:DescribeSubnets",
                    "ec2:DeleteKeyPair",
                    "ec2:AttachVolume",
                    "autoscaling:DescribeAutoScalingInstances",
                    "ec2:DeregisterImage",
                    "ec2:DeleteSnapshot",
                    "ec2:DescribeRegions",
                    "ec2:ModifyImageAttribute",
                    "elasticloadbalancing:DescribeListeners",
                    "ec2:CreateSecurityGroup",
                    "ec2:CreateSnapshot",
                    "elasticloadbalancing:DescribeListenerCertificates",
                    "ec2:ModifyInstanceAttribute",
                    "elasticloadbalancing:DescribeSSLPolicies",
                    "ec2:TerminateInstances",
                    "elasticloadbalancing:DescribeTags",
                    "ec2:DescribeTags",
                    "ec2:DescribeLaunchTemplateVersions",
                    "ec2:DescribeSecurityGroups",
                    "ec2:DescribeImages",
                    "ec2:DeleteSecurityGroup",
                    "elasticloadbalancing:DescribeTargetHealth",
                    "elasticloadbalancing:DescribeTargetGroups"
                ],
                "Resource": "*"
            },
            {
                "Sid": "2018121702",
                "Effect": "Allow",
                "Action": [
                    "ssm:PutParameter",
                    "ssm:GetParameter",
                    "autoscaling:UpdateAutoScalingGroup",
                    "ec2:ModifyLaunchTemplate",
                    "ec2:CreateLaunchTemplateVersion",
                    "autoscaling:AttachLoadBalancerTargetGroups"
                ],
                "Resource": [
                    "arn:aws:autoscaling:*:*:autoScalingGroup:*:autoScalingGroupName/*",
                    "arn:aws:ssm:eu-west-1:[YOUR_ACCOUNT]:parameter/deploying-status-*",
                    "arn:aws:ec2:*:*:launch-template/*"
                ]
            }
        ]
    }

Afraz Ahmadzadeh's avatar
Afraz Ahmadzadeh committed
115
## A Note on Role Assumption
116

Afraz Ahmadzadeh's avatar
Afraz Ahmadzadeh committed
117 118
Akinaka uses IAM roles to gain access into multiple accounts. Most commands require you to specify a list of roles you wish to perform a task for, and that role must have the [sts:AssumeRole](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_temp_control-access_enable-create.html) permission. This is not only good security, it's helpful for ensuring you're doing things to the accounts you think you're doing things for ;)

Afraz Ahmadzadeh's avatar
Afraz Ahmadzadeh committed
119
## Deploys
120

Afraz Ahmadzadeh's avatar
Afraz Ahmadzadeh committed
121 122 123 124 125
Done with the `update` parent command, and then the `asg` and `targetgroup` subcommands (`update targetgroup` is only needed for blue/green deploys).

Example:

    # For standalone ASGs (not blue/green)
Afraz Ahmadzadeh's avatar
Afraz Ahmadzadeh committed
126
    akinaka update \
127 128 129 130 131
      --region eu-west-1 \
      --role-arn arn:aws:iam::123456789100:role/management_assumable \
    asg \
      --asg workers \
      --ami ami-000000
Afraz Ahmadzadeh's avatar
Afraz Ahmadzadeh committed
132 133

    # For blue/green ASGs
Afraz Ahmadzadeh's avatar
Afraz Ahmadzadeh committed
134
    akinaka update \
135 136 137 138 139 140 141
      --region eu-west-1 \
      --role-arn arn:aws:iam::123456789100:role/management_assumable \
    asg \
      --lb lb-asg-ext \
      --ami ami-000000

    # For blue/green ASGs with multiple Target Groups behind the same ALB
Afraz Ahmadzadeh's avatar
Afraz Ahmadzadeh committed
142
    akinaka update \
143 144 145 146 147
      --region eu-west-1 \
      --role-arn arn:aws:iam::123456789100:role/management_assumable \
    asg \
      --target-group application-1a \
      --ami ami-000000
Afraz Ahmadzadeh's avatar
Afraz Ahmadzadeh committed
148 149 150 151 152 153 154

For blue/green deploys, the next step is to check the health of your new ASG.
For the purposes of Gitlab CI/CD pipelines, this will be printed out as the only
output, so that it can be used in the next job.

Once the new ASG is confirmed to be working as expected:

Afraz Ahmadzadeh's avatar
Afraz Ahmadzadeh committed
155
    akinaka update --region eu-west-1 --role-arn arn:aws:iam::123456789100:role/management_assumable asg --new blue
Afraz Ahmadzadeh's avatar
Afraz Ahmadzadeh committed
156 157 158 159 160 161 162 163

The value of `--role-arn` is used to assume a role in the target account with enough
permissions to perform the actions of modifying ASGs and Target Groups. As such,
`akinaka` is able to do cross-account deploys. It will deliberately error if you
do not supply an IAM Role ARN, in order to ensure you are deploying to the account
you think you are.

## Cleanups
164

165
Currently AMI, EBS, and RDS snapshot cleanups are supported.
Afraz Ahmadzadeh's avatar
Afraz Ahmadzadeh committed
166 167 168 169 170

Common option:

`--role-arns` is a space separated list of IAM ARNs that can be assumed by the token you are using
to run this command. The AMIs for the running instances found in these accounts will not be deleted. Not to be confused with `--role-arn`, accepted for the `update` parent command, for deploys.
Afraz Ahmadzadeh's avatar
Afraz Ahmadzadeh committed
171 172

### AMIs
173

Afraz Ahmadzadeh's avatar
Afraz Ahmadzadeh committed
174 175 176 177 178 179
Cleans up AMIs and their snapshots based on a specified retention period, and deduced AMI usage (will
not delete AMIs that are currently in use). You can optionally specify an AMI name pattern, and it will
keep the latest version of all the AMIs it finds for it.

Usage:

Afraz Ahmadzadeh's avatar
Afraz Ahmadzadeh committed
180
    akinaka cleanup \
Afraz Ahmadzadeh's avatar
Afraz Ahmadzadeh committed
181 182 183 184 185
        --region eu-west-1 \
        --role-arns "arn:aws:iam::198765432100:role/management_assumable arn:aws:iam::123456789100:role/management_assumable" \
        ami \
            --exceptional-amis cib-base-image-*
            --retention 7
Afraz Ahmadzadeh's avatar
Afraz Ahmadzadeh committed
186 187 188 189 190 191 192 193 194 195 196 197 198 199

The above will delete all AMIs and their snapshots, _except for those which:_

1. Are younger than 7 days AND
2. Are not in use by AWS accounts "123456789100" or "198765432100" AND
3. WHERE the AMI name matches the pattern "cib-base-image-*", there is more than one match AND it is the oldest one

`--exceptional-amis` is a space seperated list of exact names or patterns for which to keep the latest
version of an AMI for. For example, the pattern "cib-base-image-*" will match with normal globbing, and
if there is more than one match, only the latest one will not be deleted (else there is no effect).

`--retention` is the retention period you want to exclude from deletion. For example; `--retention 7`
will keep all AMIs found within 7 days, if they are not in the `--exceptional-amis` list.

Afraz Ahmadzadeh's avatar
Afraz Ahmadzadeh committed
200
### EBS Volumes
201

Afraz Ahmadzadeh's avatar
Afraz Ahmadzadeh committed
202 203
Delete all EBS volumes that are not attached to an instance (stopped or not):

Afraz Ahmadzadeh's avatar
Afraz Ahmadzadeh committed
204
    akinaka cleanup \
Afraz Ahmadzadeh's avatar
Afraz Ahmadzadeh committed
205 206 207 208
        --region eu-west-1 \
        --role-arns "arn:aws:iam::198765432100:role/management_assumable arn:aws:iam::123456789100:role/management_assumable" \
        ebs

209 210 211 212
### RDS Snapshots

    This will delete all snapshots tagged "akinaka-made":
    
Afraz Ahmadzadeh's avatar
Afraz Ahmadzadeh committed
213
    akinaka cleanup \
214 215 216
        --not-dry-run \
        --region eu-west-1 \
        --role-arns "arn:aws:iam::876521782800:role/OlinDataAssumedAdministrator" \
217 218
        rds \
            --tags "akinaka-made"
219

220
## RDS
221

222 223 224
Perform often necessary but complex tasks with RDS.

### Copy
225

226 227
Copy encrypted RDS instances between accounts:

Afraz Ahmadzadeh's avatar
Afraz Ahmadzadeh committed
228
    akinaka copy --region eu-west-1 \
Afraz Ahmadzadeh's avatar
Afraz Ahmadzadeh committed
229 230 231 232 233 234 235 236
        rds \
            --source-role-arn arn:aws:iam::198765432100:role/management_assumable \
            --target-role-arn arn:aws:iam::123456789100:role/management_assumable \
            --snapshot-style running_instance \
            --source-instance-name DB_FROM_ACCOUNT_198765432100 \
            --target-instance-name DB_FROM_ACCOUNT_123456789100 \
            --target-security-group SECURITY_GROUP_OF_TARGET_RDS \
            --target-db-subnet SUBNET_OF_TARGET_RDS \
237

Choon Ming Goh's avatar
Choon Ming Goh committed
238 239
`--region` is optional because it will default to the environment variable `AWS_DEFAULT_REGION`.

Afraz Ahmadzadeh's avatar
Afraz Ahmadzadeh committed
240
## Container
241

Afraz Ahmadzadeh's avatar
Afraz Ahmadzadeh committed
242 243
Limited functionality for interactive with EKS and ECR. At the moment it's just getting a docker login via an assumed role to another assumed role:

Afraz Ahmadzadeh's avatar
Afraz Ahmadzadeh committed
244
    akinaka container --region eu-west-1 --role-arn arn:aws:iam::0123456789:role/registry-rw get-ecr-login --registry 0123456789
Afraz Ahmadzadeh's avatar
Afraz Ahmadzadeh committed
245 246 247

The above will assume the role `arn:aws:iam::0123456789:role/registry-rw` in the account with the registry, and spit out a `docker login` line for you to use — exactly like `aws ecr get-login`, but working for assumed roles.

Afraz Ahmadzadeh's avatar
Afraz Ahmadzadeh committed
248
## Billing
249

250 251
Get a view of your daily AWS estimated bill for the x number of days. Defaults to today's estimated bill.

Afraz Ahmadzadeh's avatar
Afraz Ahmadzadeh committed
252
    akinaka reporting --region us-east-1 \
253
      --role-arn arn:aws:iam::1234567890:role/billing_assumerole \
Afraz Ahmadzadeh's avatar
Afraz Ahmadzadeh committed
254
      bill-estimates --from-days-ago 1
255 256 257 258 259 260 261 262 263 264 265 266 267 268

Example output:

    Today's estimated bill
    +------------+-----------+
    | Date       | Total     |
    |------------+-----------|
    | 2019-03-14 | USD 13.93 |
    +------------+-----------+

You can specify any integer value to the `--days-ago` flag. It's optional. Default value set for today (current day).

You can specify any region to the `--region` flag.

Afraz Ahmadzadeh's avatar
Afraz Ahmadzadeh committed
269
## Contributing
270

Afraz Ahmadzadeh's avatar
Afraz Ahmadzadeh committed
271
Modules can be added easily by simply dropping them in and adding an entry into `akinaka` to include them, and some `click` code in their `__init__` (or elsewhere that's loaded, but this is the cleanest way).
Afraz Ahmadzadeh's avatar
Afraz Ahmadzadeh committed
272

Afraz Ahmadzadeh's avatar
Afraz Ahmadzadeh committed
273
For example, given a module called `akinaka_moo`, and a single command and file called `moo`, add these two lines in the appropriate places of `akinaka`:
Afraz Ahmadzadeh's avatar
Afraz Ahmadzadeh committed
274

Afraz Ahmadzadeh's avatar
Afraz Ahmadzadeh committed
275 276
    from akinaka_update.commands import moo as moo_commands
    cli.add_command(moo_commands)
Afraz Ahmadzadeh's avatar
Afraz Ahmadzadeh committed
277

Afraz Ahmadzadeh's avatar
Afraz Ahmadzadeh committed
278
and the following in the module's `commands.py`:
Afraz Ahmadzadeh's avatar
Afraz Ahmadzadeh committed
279 280 281 282 283 284 285

    @click.group()
    @click.option("--make-awesome", help="The way in which to make moo awesome")
    def moo(make_awesome):
        import .moo
        # YOUR CODE USING THE MOO MODULE

Choon Ming Goh's avatar
Choon Ming Goh committed
286
Adding commands that need subcommands isn't too different, but you might want to take a look at the already present examples of `update` and `cleanup`.