README.md 18.8 KB
Newer Older
Afraz Ahmadzadeh's avatar
Afraz Ahmadzadeh committed
1
# Akinaka
2

Afraz Ahmadzadeh's avatar
Afraz Ahmadzadeh committed
3 4
This is a general all-purpose tool for managing things in AWS that Terraform is not responsible for -- you can think of it as an extension to the `aws` CLI.

5
At the moment it only does three things; blue/green deploys for plugging into Gitlab, AMI cleanups, and RDS copies to other accounts.
Afraz Ahmadzadeh's avatar
Afraz Ahmadzadeh committed
6

Afraz Ahmadzadeh's avatar
Afraz Ahmadzadeh committed
7 8 9 10 11 12 13 14
- [Akinaka](#akinaka)
  - [Installation](#installation)
  - [Requirements and Presumptions](#requirements-and-presumptions)
  - [A Note on Role Assumption](#a-note-on-role-assumption)
  - [Deploys](#deploys)
  - [Cleanups](#cleanups)
    - [AMIs](#amis)
    - [EBS Volumes](#ebs-volumes)
15
    - [RDS Snapshots](#rds-snapshots)
Afraz Ahmadzadeh's avatar
Afraz Ahmadzadeh committed
16 17
  - [RDS](#rds)
    - [Copy](#copy)
Afraz Ahmadzadeh's avatar
Afraz Ahmadzadeh committed
18
  - [Disaster Recovery](#disaster-recovery)
Afraz Ahmadzadeh's avatar
Afraz Ahmadzadeh committed
19
    - [Transfer](#transfer)
Afraz Ahmadzadeh's avatar
Afraz Ahmadzadeh committed
20 21 22
  - [Container](#container)
  - [Billing](#billing)
  - [Contributing](#contributing)
Afraz Ahmadzadeh's avatar
Afraz Ahmadzadeh committed
23 24 25

## Installation

Afraz Ahmadzadeh's avatar
Afraz Ahmadzadeh committed
26
    pip3 install akinaka
Afraz Ahmadzadeh's avatar
Afraz Ahmadzadeh committed
27

28 29 30 31 32 33 34 35
## Requirements and Presumptions

Format of ASG names: "whatever-you-like*-blue/green*" — the part in bold is necessary, i.e. you must have two ASGs, one ending with "-blue" and one ending with "-green".

The following permissions are necessary for the IAM role / user that will be running Akinaka:

    sts:AssumeRole

Afraz Ahmadzadeh's avatar
Afraz Ahmadzadeh committed
36
_NOTE: Going forward, IAM policies will be listed separately for their respective subcommands (as is already the case for [Transfer](#transfer)). For now however, the following single catch-all policy can be used, attach it to the IAM profile that Akinaka will be assuming:_
37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116

    {
        "Version": "2012-10-17",
        "Statement": [
            {
                "Sid": "2018121701",
                "Effect": "Allow",
                "Action": [
                    "ec2:AuthorizeSecurityGroupIngress",
                    "ec2:DescribeInstances",
                    "ec2:CreateKeyPair",
                    "ec2:CreateImage",
                    "ec2:CopyImage",
                    "ec2:DescribeSnapshots",
                    "elasticloadbalancing:DescribeLoadBalancers",
                    "ec2:DeleteVolume",
                    "ec2:ModifySnapshotAttribute",
                    "autoscaling:DescribeAutoScalingGroups",
                    "ec2:DescribeVolumes",
                    "ec2:DetachVolume",
                    "ec2:DescribeLaunchTemplates",
                    "ec2:CreateTags",
                    "ec2:RegisterImage",
                    "autoscaling:DetachLoadBalancerTargetGroups",
                    "ec2:RunInstances",
                    "ec2:StopInstances",
                    "ec2:CreateVolume",
                    "autoscaling:AttachLoadBalancerTargetGroups",
                    "elasticloadbalancing:DescribeLoadBalancerAttributes",
                    "ec2:GetPasswordData",
                    "elasticloadbalancing:DescribeTargetGroupAttributes",
                    "elasticloadbalancing:DescribeAccountLimits",
                    "ec2:DescribeImageAttribute",
                    "elasticloadbalancing:DescribeRules",
                    "ec2:DescribeSubnets",
                    "ec2:DeleteKeyPair",
                    "ec2:AttachVolume",
                    "autoscaling:DescribeAutoScalingInstances",
                    "ec2:DeregisterImage",
                    "ec2:DeleteSnapshot",
                    "ec2:DescribeRegions",
                    "ec2:ModifyImageAttribute",
                    "elasticloadbalancing:DescribeListeners",
                    "ec2:CreateSecurityGroup",
                    "ec2:CreateSnapshot",
                    "elasticloadbalancing:DescribeListenerCertificates",
                    "ec2:ModifyInstanceAttribute",
                    "elasticloadbalancing:DescribeSSLPolicies",
                    "ec2:TerminateInstances",
                    "elasticloadbalancing:DescribeTags",
                    "ec2:DescribeTags",
                    "ec2:DescribeLaunchTemplateVersions",
                    "ec2:DescribeSecurityGroups",
                    "ec2:DescribeImages",
                    "ec2:DeleteSecurityGroup",
                    "elasticloadbalancing:DescribeTargetHealth",
                    "elasticloadbalancing:DescribeTargetGroups"
                ],
                "Resource": "*"
            },
            {
                "Sid": "2018121702",
                "Effect": "Allow",
                "Action": [
                    "ssm:PutParameter",
                    "ssm:GetParameter",
                    "autoscaling:UpdateAutoScalingGroup",
                    "ec2:ModifyLaunchTemplate",
                    "ec2:CreateLaunchTemplateVersion",
                    "autoscaling:AttachLoadBalancerTargetGroups"
                ],
                "Resource": [
                    "arn:aws:autoscaling:*:*:autoScalingGroup:*:autoScalingGroupName/*",
                    "arn:aws:ssm:eu-west-1:[YOUR_ACCOUNT]:parameter/deploying-status-*",
                    "arn:aws:ec2:*:*:launch-template/*"
                ]
            }
        ]
    }

Afraz Ahmadzadeh's avatar
Afraz Ahmadzadeh committed
117
## A Note on Role Assumption
118

Afraz Ahmadzadeh's avatar
Afraz Ahmadzadeh committed
119 120
Akinaka uses IAM roles to gain access into multiple accounts. Most commands require you to specify a list of roles you wish to perform a task for, and that role must have the [sts:AssumeRole](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_temp_control-access_enable-create.html) permission. This is not only good security, it's helpful for ensuring you're doing things to the accounts you think you're doing things for ;)

Afraz Ahmadzadeh's avatar
Afraz Ahmadzadeh committed
121
## Deploys
122

Afraz Ahmadzadeh's avatar
Afraz Ahmadzadeh committed
123 124 125 126 127
Done with the `update` parent command, and then the `asg` and `targetgroup` subcommands (`update targetgroup` is only needed for blue/green deploys).

Example:

    # For standalone ASGs (not blue/green)
Afraz Ahmadzadeh's avatar
Afraz Ahmadzadeh committed
128
    akinaka update \
129 130 131 132 133
      --region eu-west-1 \
      --role-arn arn:aws:iam::123456789100:role/management_assumable \
    asg \
      --asg workers \
      --ami ami-000000
Afraz Ahmadzadeh's avatar
Afraz Ahmadzadeh committed
134 135

    # For blue/green ASGs
Afraz Ahmadzadeh's avatar
Afraz Ahmadzadeh committed
136
    akinaka update \
137 138 139 140 141 142 143
      --region eu-west-1 \
      --role-arn arn:aws:iam::123456789100:role/management_assumable \
    asg \
      --lb lb-asg-ext \
      --ami ami-000000

    # For blue/green ASGs with multiple Target Groups behind the same ALB
Afraz Ahmadzadeh's avatar
Afraz Ahmadzadeh committed
144
    akinaka update \
145 146 147 148 149
      --region eu-west-1 \
      --role-arn arn:aws:iam::123456789100:role/management_assumable \
    asg \
      --target-group application-1a \
      --ami ami-000000
Afraz Ahmadzadeh's avatar
Afraz Ahmadzadeh committed
150 151 152 153 154 155 156

For blue/green deploys, the next step is to check the health of your new ASG.
For the purposes of Gitlab CI/CD pipelines, this will be printed out as the only
output, so that it can be used in the next job.

Once the new ASG is confirmed to be working as expected:

Afraz Ahmadzadeh's avatar
Afraz Ahmadzadeh committed
157
    akinaka update --region eu-west-1 --role-arn arn:aws:iam::123456789100:role/management_assumable asg --new blue
Afraz Ahmadzadeh's avatar
Afraz Ahmadzadeh committed
158 159 160 161 162 163 164 165

The value of `--role-arn` is used to assume a role in the target account with enough
permissions to perform the actions of modifying ASGs and Target Groups. As such,
`akinaka` is able to do cross-account deploys. It will deliberately error if you
do not supply an IAM Role ARN, in order to ensure you are deploying to the account
you think you are.

## Cleanups
166

167
Currently AMI, EBS, and RDS snapshot cleanups are supported.
Afraz Ahmadzadeh's avatar
Afraz Ahmadzadeh committed
168 169 170 171 172

Common option:

`--role-arns` is a space separated list of IAM ARNs that can be assumed by the token you are using
to run this command. The AMIs for the running instances found in these accounts will not be deleted. Not to be confused with `--role-arn`, accepted for the `update` parent command, for deploys.
Afraz Ahmadzadeh's avatar
Afraz Ahmadzadeh committed
173 174

### AMIs
175

Afraz Ahmadzadeh's avatar
Afraz Ahmadzadeh committed
176 177 178 179 180 181
Cleans up AMIs and their snapshots based on a specified retention period, and deduced AMI usage (will
not delete AMIs that are currently in use). You can optionally specify an AMI name pattern, and it will
keep the latest version of all the AMIs it finds for it.

Usage:

Afraz Ahmadzadeh's avatar
Afraz Ahmadzadeh committed
182
    akinaka cleanup \
Afraz Ahmadzadeh's avatar
Afraz Ahmadzadeh committed
183 184 185 186 187
        --region eu-west-1 \
        --role-arns "arn:aws:iam::198765432100:role/management_assumable arn:aws:iam::123456789100:role/management_assumable" \
        ami \
            --exceptional-amis cib-base-image-*
            --retention 7
Afraz Ahmadzadeh's avatar
Afraz Ahmadzadeh committed
188 189 190 191 192 193 194 195 196 197 198 199 200 201

The above will delete all AMIs and their snapshots, _except for those which:_

1. Are younger than 7 days AND
2. Are not in use by AWS accounts "123456789100" or "198765432100" AND
3. WHERE the AMI name matches the pattern "cib-base-image-*", there is more than one match AND it is the oldest one

`--exceptional-amis` is a space seperated list of exact names or patterns for which to keep the latest
version of an AMI for. For example, the pattern "cib-base-image-*" will match with normal globbing, and
if there is more than one match, only the latest one will not be deleted (else there is no effect).

`--retention` is the retention period you want to exclude from deletion. For example; `--retention 7`
will keep all AMIs found within 7 days, if they are not in the `--exceptional-amis` list.

Afraz Ahmadzadeh's avatar
Afraz Ahmadzadeh committed
202
### EBS Volumes
203

Afraz Ahmadzadeh's avatar
Afraz Ahmadzadeh committed
204 205
Delete all EBS volumes that are not attached to an instance (stopped or not):

Afraz Ahmadzadeh's avatar
Afraz Ahmadzadeh committed
206
    akinaka cleanup \
Afraz Ahmadzadeh's avatar
Afraz Ahmadzadeh committed
207 208 209 210
        --region eu-west-1 \
        --role-arns "arn:aws:iam::198765432100:role/management_assumable arn:aws:iam::123456789100:role/management_assumable" \
        ebs

211 212 213 214
### RDS Snapshots

    This will delete all snapshots tagged "akinaka-made":
    
Afraz Ahmadzadeh's avatar
Afraz Ahmadzadeh committed
215
    akinaka cleanup \
216 217 218
        --not-dry-run \
        --region eu-west-1 \
        --role-arns "arn:aws:iam::876521782800:role/OlinDataAssumedAdministrator" \
219 220
        rds \
            --tags "akinaka-made"
221

222
## RDS
223

224 225 226
Perform often necessary but complex tasks with RDS.

### Copy
227

228 229
Copy encrypted RDS instances between accounts:

Afraz Ahmadzadeh's avatar
Afraz Ahmadzadeh committed
230
    akinaka copy --region eu-west-1 \
Afraz Ahmadzadeh's avatar
Afraz Ahmadzadeh committed
231 232 233 234 235 236 237 238
        rds \
            --source-role-arn arn:aws:iam::198765432100:role/management_assumable \
            --target-role-arn arn:aws:iam::123456789100:role/management_assumable \
            --snapshot-style running_instance \
            --source-instance-name DB_FROM_ACCOUNT_198765432100 \
            --target-instance-name DB_FROM_ACCOUNT_123456789100 \
            --target-security-group SECURITY_GROUP_OF_TARGET_RDS \
            --target-db-subnet SUBNET_OF_TARGET_RDS \
239

Choon Ming Goh's avatar
Choon Ming Goh committed
240 241
`--region` is optional because it will default to the environment variable `AWS_DEFAULT_REGION`.

Afraz Ahmadzadeh's avatar
Afraz Ahmadzadeh committed
242 243 244 245 246 247 248 249 250 251 252 253 254 255 256
## Disaster Recovery

Akinaka has limited functionality for backing up and restoring data for use in disaster recovery.

### Transfer

Transfer data from S3, RDS, and RDS Aurora into a backup account:

    akinaka dr \
      --region eu-west-1 \
      --source-role-arn arn:aws:iam::[LIVE_ACCOUNT_ID]:role/[ROLE_NAME] \
      --destination-role-arn arn:aws:iam::[BACKUP_ACCOUNT_ID]:role/[ROLE_NAME] \
      transfer \
        --service s3

257 258 259
Omitting `--service` will include all supported services.

You can optionally specify the name of the instance to transfer with `--names` in a comma separated list, e.g. `--names 'database-1, database-2`. This can be for either RDS instances, or S3 buckets, but not both at the same time. Future versions may remove `--service` and replace it with a subcommand instead, i.e. `akinaka dr transfer rds`, so that those service can have `--names` to themselves.
Afraz Ahmadzadeh's avatar
Afraz Ahmadzadeh committed
260

Afraz Ahmadzadeh's avatar
Afraz Ahmadzadeh committed
261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278
A further limitation is that only a single region can be handled at a time for S3 buckets. If you wish to backup all S3 buckets in an account, and they are in different regions, you will have to specify them per run, using the appropriate region each time. Future versions will work the bucket regions out automatically, and remove this limitation.

Akinaka must be run from either an account or instance profile which can use sts:assume to assume both the `source-role-arn` and `destination-role-arn`. This is true even if you are running on the account that `destination-role-arn` is on. You will therefore need this policy attached to the user/role that's doing the assuming:

    {
        "Version": "2012-10-17",
        "Statement": [
            {
                "Sid": "akinakaassume",
                "Effect": "Allow",
                "Action": "sts:AssumeRole",
                "Resource": [
                    "arn:aws:iam::[DESTINATION_ACCOUNT]:role/[ROLE_TO_ASSUME]",
                    "arn:aws:iam::[SOURCE_ACCOUNT]:role/[ROLE_TO_ASSUME]"
                ]
            }
        ]
    }
Afraz Ahmadzadeh's avatar
Afraz Ahmadzadeh committed
279

280 281
**Note:** A period of 4 hours (469822 seconds) is hardcoded into the sts:assume call made in the RDS snapshot class, since snapshot creation can take a very long time. This must therefore be the minimum value for the role's `max-session-duration`.

Afraz Ahmadzadeh's avatar
Afraz Ahmadzadeh committed
282
The following policy is needed for usage of this subcommand, attach it to the role you'll be assuming:
Afraz Ahmadzadeh's avatar
Afraz Ahmadzadeh committed
283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343

    {
        "Version": "2012-10-17",
        "Statement": [
            {
                "Sid": "KMSEncrypt",
                "Effect": "Allow",
                "Action": [
                    "kms:GetPublicKey",
                    "kms:ImportKeyMaterial",
                    "kms:Decrypt",
                    "kms:UntagResource",
                    "kms:PutKeyPolicy",
                    "kms:GenerateDataKeyWithoutPlaintext",
                    "kms:Verify",
                    "kms:ListResourceTags",
                    "kms:GenerateDataKeyPair",
                    "kms:GetParametersForImport",
                    "kms:TagResource",
                    "kms:Encrypt",
                    "kms:GetKeyRotationStatus",
                    "kms:ReEncryptTo",
                    "kms:DescribeKey",
                    "kms:Sign",
                    "kms:CreateGrant",
                    "kms:ListKeyPolicies",
                    "kms:UpdateKeyDescription",
                    "kms:ListRetirableGrants",
                    "kms:GetKeyPolicy",
                    "kms:GenerateDataKeyPairWithoutPlaintext",
                    "kms:ReEncryptFrom",
                    "kms:RetireGrant",
                    "kms:ListGrants",
                    "kms:UpdateAlias",
                    "kms:RevokeGrant",
                    "kms:GenerateDataKey",
                    "kms:CreateAlias"
                ],
                "Resource": [
                    "arn:aws:kms:*:*:alias/*",
                    "arn:aws:kms:*:*:key/*"
                ]
            },
            {
                "Sid": "KMSCreate",
                "Effect": "Allow",
                "Action": [
                    "kms:DescribeCustomKeyStores",
                    "kms:ListKeys",
                    "kms:GenerateRandom",
                    "kms:UpdateCustomKeyStore",
                    "kms:ListAliases",
                    "kms:CreateKey",
                    "kms:ConnectCustomKeyStore",
                    "kms:CreateCustomKeyStore"
                ],
                "Resource": "*"
            }
        ]
    }

Afraz Ahmadzadeh's avatar
Afraz Ahmadzadeh committed
344
The following further policies need to be attached to the assume roles to backup each service:
Afraz Ahmadzadeh's avatar
Afraz Ahmadzadeh committed
345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371

#### RDS / RDS Aurora

    {
        "Version": "2012-10-17",
        "Statement": [
            {
                "Sid": "RDSBackup",
                "Effect": "Allow",
                "Action": [
                    "rds:DescribeDBClusterSnapshotAttributes",
                    "rds:AddTagsToResource",
                    "rds:RestoreDBClusterFromSnapshot",
                    "rds:DescribeDBSnapshots",
                    "rds:DescribeGlobalClusters",
                    "rds:CopyDBSnapshot",
                    "rds:CopyDBClusterSnapshot",
                    "rds:DescribeDBSnapshotAttributes",
                    "rds:ModifyDBSnapshot",
                    "rds:ListTagsForResource",
                    "rds:CreateDBSnapshot",
                    "rds:DescribeDBClusterSnapshots",
                    "rds:DescribeDBInstances",
                    "rds:CreateDBClusterSnapshot",
                    "rds:ModifyDBClusterSnapshotAttribute",
                    "rds:ModifyDBSnapshotAttribute",
                    "rds:DescribeDBClusters",
372 373
                    "rds:DeleteDBSnapshot",
                    "rds:DeleteDBClusterSnapshot"
Afraz Ahmadzadeh's avatar
Afraz Ahmadzadeh committed
374 375 376 377 378 379 380 381 382 383 384 385 386 387 388 389 390 391 392 393 394 395 396 397 398 399 400 401 402 403 404 405 406 407 408 409 410 411 412 413 414 415 416 417 418
                ],
                "Resource": "*"
            }
        ]
    }

#### S3

    {
        "Version": "2012-10-17",
        "Statement": [
            {
                "Sid": "S3RW",
                "Effect": "Allow",
                "Action": [
                    "s3:ListBucketMultipartUploads",
                    "s3:GetObjectRetention",
                    "s3:GetObjectVersionTagging",
                    "s3:ListBucketVersions",
                    "s3:CreateBucket",
                    "s3:ListBucket",
                    "s3:GetBucketVersioning",
                    "s3:GetBucketAcl",
                    "s3:GetObjectAcl",
                    "s3:GetObject",
                    "s3:GetEncryptionConfiguration",
                    "s3:ListAllMyBuckets",
                    "s3:PutLifecycleConfiguration",
                    "s3:GetObjectVersionAcl",
                    "s3:GetObjectTagging",
                    "s3:GetObjectVersionForReplication",
                    "s3:HeadBucket",
                    "s3:GetBucketLocation",
                    "s3:PutBucketVersioning",
                    "s3:GetObjectVersion",
                    "s3:PutObject",
                    "s3:PutObjectAcl",
                    "s3:PutEncryptionConfiguration",
                    "s3:PutBucketPolicy"
                ],
                "Resource": "*"
            }
        ]
    }

Afraz Ahmadzadeh's avatar
Afraz Ahmadzadeh committed
419
## Container
420

Afraz Ahmadzadeh's avatar
Afraz Ahmadzadeh committed
421 422
Limited functionality for interactive with EKS and ECR. At the moment it's just getting a docker login via an assumed role to another assumed role:

Afraz Ahmadzadeh's avatar
Afraz Ahmadzadeh committed
423
    akinaka container --region eu-west-1 --role-arn arn:aws:iam::0123456789:role/registry-rw get-ecr-login --registry 0123456789
Afraz Ahmadzadeh's avatar
Afraz Ahmadzadeh committed
424 425 426

The above will assume the role `arn:aws:iam::0123456789:role/registry-rw` in the account with the registry, and spit out a `docker login` line for you to use — exactly like `aws ecr get-login`, but working for assumed roles.

Afraz Ahmadzadeh's avatar
Afraz Ahmadzadeh committed
427
## Billing
428

429 430
Get a view of your daily AWS estimated bill for the x number of days. Defaults to today's estimated bill.

Afraz Ahmadzadeh's avatar
Afraz Ahmadzadeh committed
431
    akinaka reporting --region us-east-1 \
432
      --role-arn arn:aws:iam::1234567890:role/billing_assumerole \
Afraz Ahmadzadeh's avatar
Afraz Ahmadzadeh committed
433
      bill-estimates --from-days-ago 1
434 435 436 437 438 439 440 441 442 443 444 445 446 447

Example output:

    Today's estimated bill
    +------------+-----------+
    | Date       | Total     |
    |------------+-----------|
    | 2019-03-14 | USD 13.93 |
    +------------+-----------+

You can specify any integer value to the `--days-ago` flag. It's optional. Default value set for today (current day).

You can specify any region to the `--region` flag.

Afraz Ahmadzadeh's avatar
Afraz Ahmadzadeh committed
448
## Contributing
449

Afraz Ahmadzadeh's avatar
Afraz Ahmadzadeh committed
450
Modules can be added easily by simply dropping them in and adding an entry into `akinaka` to include them, and some `click` code in their `__init__` (or elsewhere that's loaded, but this is the cleanest way).
Afraz Ahmadzadeh's avatar
Afraz Ahmadzadeh committed
451

Afraz Ahmadzadeh's avatar
Afraz Ahmadzadeh committed
452
For example, given a module called `akinaka_moo`, and a single command and file called `moo`, add these two lines in the appropriate places of `akinaka`:
Afraz Ahmadzadeh's avatar
Afraz Ahmadzadeh committed
453

Afraz Ahmadzadeh's avatar
Afraz Ahmadzadeh committed
454 455
    from akinaka_update.commands import moo as moo_commands
    cli.add_command(moo_commands)
Afraz Ahmadzadeh's avatar
Afraz Ahmadzadeh committed
456

Afraz Ahmadzadeh's avatar
Afraz Ahmadzadeh committed
457
and the following in the module's `commands.py`:
Afraz Ahmadzadeh's avatar
Afraz Ahmadzadeh committed
458 459 460 461 462 463 464

    @click.group()
    @click.option("--make-awesome", help="The way in which to make moo awesome")
    def moo(make_awesome):
        import .moo
        # YOUR CODE USING THE MOO MODULE

Choon Ming Goh's avatar
Choon Ming Goh committed
465
Adding commands that need subcommands isn't too different, but you might want to take a look at the already present examples of `update` and `cleanup`.