have separate name values for the source configs
name: aws
tables: [ "*" ]
skip_tables: [ "aws_inspector2_findings" ]
name: aws_findings
tables: [ "aws_inspector2_findings" ]
Got it… checking. Will let you know if this works. I’m using the API key and version 24.1.0 to implement incremental sync for the aws_inspector2_findings table.
We have the CloudQuery API key, but how do I use it to sync tables? Also, what will be the validity of the key?
Hi @concise-oryx! You use an API key as an environment variable (see the docs).
Hi @KernelKnight,
Can you let me know what will be the validity for this key?
Also, I can see this issue when I run the incremental sync:
ERROR:
failed to sync records: failed to sync unmanaged client: your configuration references the following premium tables: "aws_inspector2_findings". Please run `cloudquery login` or use a valid API Key which can be generated via https://cloud.cloudquery.io to allow the sync to succeed
Is this because of the API Key or something else? I can see that inspectorv2_findings is a premium table in AWS plugin version 24.1.0.
Hey!! Any update on this?
Hi @concise-oryx,
When you create an API key, you can configure the validity. The error you’re getting could mean that you’re not setting it in the CLI environment.
When running the CLI on a Unix system (Linux/MacOS), it should look like this:
CLOUDQUERY_API_KEY=<key-value> cloudquery sync ...
I have done it. Added it like the same way.
Can it be that, as Inspector V2 is a premium table in the 24.1.0, it is not working?
Do we have anything like, API key for free tables and premium tables?
The API key should work for all tables; it’s not scoped to a specific table. I would recommend updating to the latest plugin version to ensure compatibility.
No, I am saying it like as we are using CloudQuery as an open source. Is this key only for open source tables in that version? Or can we use any premium version till some free number of row count?
The key is used for two purposes:
- You need the key to be able to download recent versions of the plugins.
- We use the key to track usage (number of rows synced) for paid tables.
We don’t track usage of free tables at the moment.
Should this key be included before a CloudQuery sync command? Like
cq_api_key = "key" cloudquery sync source.yml postgres.yml
The command should look like
CLOUDQUERY_API_KEY=<key-value> cloudquery sync source.yml postgres.yml
okay thanks
Hi @erez, I used the same command and was able to retrieve the key and have run it… But still, I am running into issues.
CLOUDQUERY_API_KEY=xxxxxxxxx: command not found
Can you let me know why this is happening?
Hi @concise-oryx, which operating system are you using? Linux, macOS, or Windows?
Ok, so you should probably reference the local binary via
CLOUDQUERY_API_KEY=<key-value> ./cloudquery sync ...
Can you let me know how to use this command in Windows?
For Windows, it would be a 2 lines command:
set CLOUDQUERY_API_KEY=value
cloudquery.exe sync ...
Thanks for your quick response. But when I am doing this sync locally in my Linux environment, I am running into these issues:
Error: failed to sync v3 source aws_xxxxxxxxxx: unexpected error from sync client receive: rpc error: code = Unavailable desc = failed to sync records: failed to sync unmanaged client: failed to create state client: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp: address unix://C:\\Users\\xxxx\\AppData\\Local\\Temp\\cq-CVfYcHqlycOCdMzE.sock: too many colons in address"
| Syncing resources... (0/-, 0 resources/hr) [0s]
Are you running on Linux or Windows? The error you shared has a Windows path. Since this is a different issue from the API key one, I recommend opening a bug report via GitHub.
I am running it in a bash terminal. Is it Linux?
Ah yes, I think you’re hitting the bug in this GitHub issue. Can you share your configuration spec on that issue?