Aws source plugin for org admin_account issue after updating to v24.3.2

CloudQuery Plugins
1

Did something change in the AWS source plugin for org/admin_account I’m not seeing? This is fine in v23.1.0:

spec:
  org:
    admin_account:
      role_arn: arn:aws:iam::1111111111:role/cloud-query
    member_role_name: cloud-query

But in v24.3.2 I get:

{
  "level": "error",
  "module": "cli",
  "error": "failed to sync v3 source aws: failed to init source aws: jsonschema: /org/admin_account does not validate with https://github.com/cloudquery/cloudquery/plugins/source/aws/client/spec/spec#/$ref/properties/org/oneOf/0/$ref/properties/admin_account/oneOf/0/$ref/required: missing properties: id",
  "time": "2024-02-22T18:56:42Z",
  "message": "exiting with error"
}

The docs don’t show anything that stands out, though. It looks like this change may have created side effects not documented to schema.json from "$ref": "#/$defs/StringFilter" to "#/$defs/Account" in the commit.

2

The docs do say the id field is required. Let me check this link.

3

v24.3.0 works without change. I don’t see anything in the changelogs to indicate this.

4

I think there was an SDK update that might have enforced that validation.

5

Configuration account != org admin_account, which is what seems off.
The docs specifically for spec.org don’t mention any changes needed or id at all, neither do the changelogs.
Seems like the change from a string to an AccountRef requires an ID that isn’t in any upgrade notes.

6

org is of type Account here, which can be found here.

7

Right, so a change is needed but nothing lines up in the changelog or the org portion of the docs.

Isn’t it clear you need to add a new property or what the property should be?

https://hub.cloudquery.io/plugins/source/cloudquery/aws/latest/versions

It doesn’t mention anything about org, admin_account, organization, etc. for a breaking change is what I’m getting at.

It’s worth updating the changelog in v24.3.1 or v24.3.2, whichever introduced it.

8

Yeah, I agree. Let me look into that.

9

I’m not sure if it’s v24.3.1 or v24.3.2 that introduced it, I’m pinned to v24.3.0.

10

I think it’s actually a CLI change that started enforcing the validation (before we would print a log warning).

This pull request should revert the CLI change. We’ll need to release it as a major bump, probably.

Thanks for reporting.

11

Thanks for the fast feedback!

12

This should be fixed in CLI v5.5.7. Do mind updating and see if that works?

13

I’m only using this through the AWS source provider, which is still v24.3.2 for the latest. If there’s a bump to that which includes 5.5.7 CLI, I can test pretty easily.

14

What do you mean by the AWS source provider? How are you running cloudquery sync?
The bug happens with a recent CLI version + latest AWS version. It’s a bug in the CLI but only appears on the latest plugins.
We’ve released a new CLI that should fix it.
I hope that clarifies things, sorry for the versions confusion :upside_down_face:

15 
19           containers:
 20             - name: cloudquery
 21               image: ghcr.io/cloudquery/cloudquery:latest
 22               imagePullPolicy: Always
 23               args: ["sync", "/config/config.yml", "--log-console", "--log-format", "json"]

I’m running it as a cron where I use a ConfigMap for the sources, which is pinned at v24.3.0. If ghcr.io/cloudquery/cloudquery:latest is bumped, then I’m guessing it should just work?

16

Yeah, it should just work if you’re using the latest from Docker.

17

I have imagePullPolicy: Always, but no change.

Error: failed to sync v3 source aws: failed to init source aws: jsonschema: '/org/admin_account' does not validate with https://github.com/cloudquery/cloudquery/plugins/source/aws/client/spec/spec#/$ref/properties/org/oneOf/0/$ref/properties/admin_account/oneOf/0/$ref/required: missing properties: 'id'
{"level":"error","module":"cli","error":"failed to sync v3 source aws: failed to init source aws: jsonschema: '/org/admin_account' does not validate with https://github.com/cloudquery/cloudquery/plugins/source/aws/client/spec/spec#/$ref/properties/org/oneOf/0/$ref/properties/admin_account/oneOf/0/$ref/required: missing properties: 'id'","time":"2024-02-22T19:58:55Z","message":"exiting with error"}

I can confirm it’s a warning if I use :5.5.7 instead of latest, so it might be a caching issue on my side.

18

Yeah, the pod probably needs to be re-created for the new image to get pulled.