Not receiving resources after running sync with new aws organizations tables

CloudQuery Plugins
1

I recently added several new tables, but didn’t get any resource after running sync.

$ more aws_organizations.yml
kind: source
spec:
  name: "aws-organizations"
  path: "cloudquery/aws"
  version: "v22.12.0"
  tables:
    - aws_organizations
    - aws_organizations_accounts
    - aws_organizations_account_parents
    - aws_organizations_delegated_services
    - aws_organizations_delegated_administrators
    - aws_organizations_organizational_units
    - aws_organizations_organizational_unit_parents
    - aws_organizations_policies
    - aws_organizations_roots
  destinations: ["postgresql"]
  spec:
    regions:
      - ap-southeast-2
    aws_debug: false
    org:
      admin_account:
        # make sure you have created local aws config, reference aws-config-<env> file
        local_profile: cloudquery
      member_role_name: cloudquery-ro

I used the assume role, built from the template.yml, and already successfully synced accounts under the whole organization with other tables.

However, I’m not sure why I’m not getting the resource from AWS Organizations itself.

I ran the sync command from a delegated administrator account, not the master (organization root) account.

2

Hey @calm-walrus - do you see any errors in your logs and what type of delegated administrator account are you running from (what AWS service(s) or organization features were delegated to that account)?

3

@stunning-dodo

Here is the error log:

2023-10-05T00:23:41Z ERR error retrieving credentials error="failed to refresh cached credentials, operation error STS: AssumeRole, https response error StatusCode: 403, RequestID: 53xxxx, api error AccessDenied: User: arn:aws:sts::<delegated_account_id>:assumed-role/cloudquery-mgmt-ro/aws-go-sdk-1xxxx is not authorized to perform: sts:AssumeRole on resource: arn:aws:iam::<master_account_id>:role/cloudquery-ro" account=<master_account_id> module=aws-src

So, it seems the official IAM CloudFormation template (to create CloudFormation StackSets) didn’t create the assume IAM role in the master account. It doesn’t create the assume role for all client accounts.

CloudFormation Template

4

Hey @calm-walrus, StackSets don’t deploy stack instances to the organization’s management account (AWS implementation).

When using StackSets, did you use Service Managed Permissions?

Alternatively, you could deploy a single CloudFormation stack into the management account with the cloudquery-ro role and the appropriate trust policy.

5

Yes, I can. Just realized this template and CloudFormation stack sets don’t cover the master account.

6

We’ll add a disclaimer to the repository to note AWS’s model for deploying StackSets and how they may not deploy to the Org management account. GitHub Issue #4

From AWS Documentation: AWS CloudFormation StackSets Getting Started - “StackSets doesn’t deploy stack instances to the organization’s management account, even if the management account is in your organization or in an OU in your organization.”

7

@stunning-dodo

Finally, I fixed the issue.

It is because I limited the regions to my country only. The debug log complained about it.

After cleaning the region limitation, I can sync the resources from the AWS organization now.

So, two changes I made:

  1. Set assume role in the master account as well.
  2. Unlimited regions.

Here is the source configuration for your reference. I created a separate source configuration file for this task.

kind: source
spec:
  name: "aws-organizations"
  path: "cloudquery/aws"
  version: "v22.12.0"
  tables:
  - aws_organizations
  - aws_organizations_accounts
  - aws_organizations_account_parents
  - aws_organizations_delegated_services
  - aws_organizations_delegated_administrators
  - aws_organizations_organizational_units
  - aws_organizations_organizational_unit_parents
  - aws_organizations_policies
  - aws_organizations_roots
  destinations: ["postgresql"]
  spec:
    # don't limit the regions when sync tables for AWS Organization
    # regions:
    #   - ap-southeast-2
    # aws_debug: false
    org:
      admin_account:
        # make sure you have created local aws config, reference aws-config-<env> file
        local_profile: cloudquery
      member_role_name: cloudquery-ro
8

Nice catch, @calm-walrus!