CloudQuery fetching public AMI details for AWS EC2 instances

CloudQuery Plugins
1

Hi everyone. First time here, looking for help.

I use CloudQuery in my organization to fetch various information from AWS. I’m facing a challenge with tables aws_ec2_instances and aws_ec2_images. I’m simply trying to build a dashboard with all our EC2 instances and the image names used for each. For custom AMIs built by us, it’s easy: I will find them in the second table. But for AMIs from Amazon or from the Marketplace, I don’t have any information and I’m stuck with just the image IDs…

Is there a way to fetch public image details with CloudQuery?

Thanks!

2

@concise-panther That sounds like a perfect use for CloudQuery! As for your question about how to get the aws_ec2_images for public images, one of the ways to do this might be to use the table_options to override the API calls being made.

3

Thanks for your feedback! I’m not familiar with table_options, but I will look into it!

4

Here is an example spec that uses the table_options to sync all of the aws-marketplace images:

kind: source
spec:
  name: "aws"
  path: "cloudquery/aws"
  version: "v23.5.0"
  skip_dependent_tables: true
  tables:
    - "aws_ec2_images"
  destinations: ["postgresql"]
  spec:
    table_options:
      aws_ec2_images:
        describe_images:
          - owners: ["aws-marketplace"]
---
kind: destination
spec:
  name: "postgresql"
  path: "cloudquery/postgresql"
  registry: "cloudquery"
  version: "v7.2.0"  
  migrate_mode: forced
  spec:
    connection_string: "postgresql://postgres:pass@localhost:5432/postgres?sslmode=disable"

And then after the sync:

select count(*), region from aws_ec2_images group by region

"count"    "region"
19282    "ap-northeast-1"
19135    "ap-northeast-2"
16682    "ap-northeast-3"
19181    "ap-south-1"
19258    "ap-southeast-1"
19239    "ap-southeast-2"
19199    "ca-central-1"
19356    "eu-central-1"
18948    "eu-north-1"
19439    "eu-west-1"
19246    "eu-west-2"
19009    "eu-west-3"
18996    "sa-east-1"
19844    "us-east-1"
19505    "us-east-2"
19397    "us-west-1"
19576    "us-west-2"
5

Thank you for providing an example config. It’s getting late here, so I will probably check it next week, but it’s definitely useful.

6

@concise-panther - Checking in on this to see if you got it working or if you need any help getting it running as you need it.