eventually fails with
Error: failed to sync v3 source aws: rpc error: code = Internal desc = failed to init plugin: failed to initialize client: error retrieving AWS credentials (see logs for details). Please verify your credentials and try again
Are there any logs after this error happens?
Loading spec(s) from /config/cloudquery/
Downloading https://github.com/cloudquery/cloudquery/releases/download/plugins-source-aws-v22.6.0/aws_linux_amd64.zip
Downloading 100% |███████████████████████████████| (59/59 MB, 206 MB/s)
Downloading 100% |█████████████████████████████| (7.6/7.6 MB, 242 MB/s)
Error: failed to sync v3 source aws: rpc error: code = Internal desc = failed to init plugin: failed to initialize client: error retrieving AWS credentials
2023-10-10T07:24:59Z INF Loading spec(s) args=["/config/cloudquery/"] module=cli
2023-10-10T07:25:02Z INF Plugin server listening address=/tmp/cq-dlvkYdeGkJVZINKq.sock module=cli
2023-10-10T07:25:03Z INF Plugin server listening address=/tmp/cq-eTnIDnIQMJgusUiL.sock module=cli
2023-10-10T07:25:04Z INF started call grpc.code=OK grpc.component=server grpc.method=GetVersions grpc.method_type=unary grpc.service=cloudquery.discovery.v1.Discovery grpc.start_time=2023-10-10T07:25:04Z grpc.time_ms=0.035 module=cli peer.address=@ protocol=grpc
2023-10-10T07:25:04Z INF finished call grpc.code=OK grpc.component=server grpc.method=GetVersions grpc.method_type=unary grpc.service=cloudquery.discovery.v1.Discovery grpc.start_time=2023-10-10T07:25:04Z grpc.time_ms=0.075 module=cli peer.address=@ protocol=grpc
2023-10-10T07:25:04Z INF started call grpc.code=OK grpc.component=server grpc.method=GetVersions grpc.method_type=unary grpc.service=cloudquery.discovery.v1 grpc.time_ms=0.018 module=cli peer.address=@ protocol=grpc
What am I missing over there @herman?
How are you storing/sourcing your AWS credentials?
We’ve had the service account role in the EKS cluster, so we’re not using the AWS credentials.
Can you run the sync with verbose logging (CloudQuery Troubleshooting Documentation) and paste the logs from the cloudquery.log file here?
Does the default IAM role attached to your EKS node have permission to assume the arn:aws:iam::34xxxxxxxxxxx7:role/FinOpsSwitch role?
yeah it does, sure will paste the logs in a moment
This is what I get
/app # ./cloudquery sync /config/cloudquery/ --log-level debug
Loading spec(s) from /config/cloudquery/
Downloading https://github.com/cloudquery/cloudquery/releases/download/plugins-source-aws-v22.6.0/aws_linux_amd64.zip
Downloading 100% |███████████████████████████████████████████████████████████████████████████████████████████████████████████| (59/59 MB, 279 MB/s)
Downloading https://github.com/cloudquery/cloudquery/releases/download/plugins-destination-postgresql-v5.0.5/postgresql_linux_amd64.zip
Downloading 100% |█████████████████████████████████████████████████████████████████████████████████████████████████████████| (7.6/7.6 MB, 268 MB/s)
Error: failed to sync v3 source aws: rpc error: code = Internal desc = failed to init plugin: failed to initialize client: error retrieving AWS credentials (see logs for details). Please verify your credentials and try again
Could you check/copy the contents of cloudquery.log as well?
I think it’s the permission issue
{
"Version": "2012-10-17",
"Statement": [
{
"Action": "sts:AssumeRole",
"Effect": "Allow",
"Resource": "arn:aws:iam::*:role/Costy"
}
]
}
So I think I am missing
sts:AssumeRoleWithWebIdentity
Btw, just wanted to be sure, we are not using any web identity, then why does it throw me this exception?
I believe it’s because EKS uses OIDC to assume roles (which AWS calls webIdentity).
@marcel @herman Still getting 403
2023-10-12T06:35:12Z ERR error retrieving credentials error="failed to refresh cached credentials, operation error STS: AssumeRole, https response error StatusCode: 403, RequestID: 0d7372c0-7b11-464f-af6d-8b9dc8bcb113, api error AccessDenied: User: arn:aws:sts::345746469657:assumed-role/FinOpsSwitch/1697092511907184560 is not authorized to perform: sts:AssumeRole on resource: arn:aws:iam::345746469657:role/FinOpsSwitch" module=aws-src
2023-10-12T06:35:12Z ERR error getting child accounts error="error retrieving AWS credentials (see logs for details). Please verify your credentials and try again" module=aws-src
2023-10-12T06:35:12Z ERR finished call grpc.code=Internal grpc.component=server grpc.error="rpc error: code = Internal desc = failed to init plugin: failed to initialize client: error retrieving AWS credentials (see logs for details). Please verify your credentials and try again" grpc.method=Init grpc.method_type=unary grpc.service=cloudquery.plugin.v3.Plugin grpc.start_time=2023-10-12T06:35:11Z grpc.time_ms=246.351 module=cli peer.address=@ protocol=grpc
My use case is I have one centralized billing account via which I can assume role and get the resources of 40 AWS accounts, and I am trying it with the below destination PostgreSQL configuration and it eventually says access denied??? Why??
destinations: ["postgresql"]
spec:
aws_debug: false
org:
admin_account:
role_arn: arn:aws:iam::345xxxxxxxxx7:role/FinOpsSwitch
member_role_name: Costy
regions:
- '*'
Are you able to view these resources using the AWS CLI by assuming the arn:aws:iam::345xxxxxxxxx7:role/FinOpsSwitch role?
Looks like I’ve figured out the issue. Actually, I was trying to assume the role of the admin account instead of the member account. Since I have the service account configured with the admin account role FinOpsSwitch, a small change in the configuration worked for me.
Thanks @marcel and @herman for your responses, appreciate it.
The expected config was:
destinations: ["postgresql"]
spec:
aws_debug: false
org:
member_role_name: Costy
regions:
- '*'