CloudQuery AWS source plugin v23 update missing column information

CloudQuery Plugins
1

Hello team,

I have a question about the CloudQuery AWS source plugin v23.0.0 update.

We were using the AWS source plugin version 22 and now we want to upgrade, but we are having difficulties due to insufficient information about the change of table structures.

With the v23.0.0 plugin update, the data in the aws_s3_buckets table was reduced from 23 columns to 11 columns and we could not find where the block_public_acls, block_public_policy, ignore_public_acls, and restrict_public_buckets columns we need were moved.

When we examined the changelogs, we found nothing but the following explanation:

v23.0.0
⚠ BREAKING CHANGES
Refactor aws_s3_buckets to use nested tables rather than that column resolvers (#15395)

From which table can we find these values for each of our AWS buckets? Or did you decide to delete these columns?

Thank you very much.

2

Hi! The answer you’re looking for is in this cq-bot comment: GitHub Comment

  • Table aws_s3_bucket_loggings was added
  • Table aws_s3_bucket_ownership_controls was added
  • Table aws_s3_bucket_policies was added
  • Table aws_s3_bucket_public_access_blocks was added
  • Table aws_s3_bucket_replications was added
  • Table aws_s3_bucket_versionings was added

So all of the removed columns were extracted into separate tables and you should be able to JOIN them to get the fields.

3

Thank you, I understand you, Kemal Abi. However, when I examined the columns of the newly added tables you showed here one by one, I still could not find a table with block_public_acls, block_public_policy, ignore_public_acls, restrict_public_buckets data. That’s why I already applied to your Discord channel.

LinkedIn’den bağlantı isteği atmıştım ama gözünden kaçtı herhalde abi. Takip ediyorum çalışmalarını, kolay gelsin, İngiltere’ye selamlar.

4

Selamlar,

Let me take a look at the table definitions and get back to you. Those columns were moved to the aws_s3_bucket_public_access_blocks table, but they are inside the public_access_block_configuration JSON object and not separate columns.

To convert your query to the new version, you need to left join using bucket_arn :grinning: (if there is no PublicAccessBlock set in a bucket, there won’t be a row in the aws_s3_bucket_public_access_blocks table).

5

Thank you, I will edit the query this way :blush:
@kemal Hi again Kemal Abi. :wave:
https://github.com/cloudquery/cloudquery/pull/15400
Again, I found this merge about aws_cloudtrail_trail_event_selectors table change in version v23.0.0, but I couldn’t find where the read_write_type and include_management_events columns were moved.
I tried to look inside the advanced_event_selectors and event_selectors JSON columns newly added to this table, but I got stuck because I don’t know the JSON structures.
Is there a page that describes the structures of JSON columns in CloudQuery tables?
If not, can you tell me where I can find the read_write_type and include_management_events columns removed from aws_cloudtrail_trail_event_selectors table?
Thank you.

6

JSON structures are directly as they come in from the AWS API. If you go here for the table: CloudQuery AWS Table and click on the docs.aws.amazon.com link, it should take you to the API page.

EventSelectors is the key under the event_selectors column, which has an array of each selector separately. To update an older query, you might need to use UNNEST() (this is for PostgreSQL) like functionality to create multiple rows for each of the array elements.

7

This is exactly what I needed, thank you for enlightening me, Kemal Abi :blush: