# Issue with adding dependent tables in CloudQuery **URL:** \ **Category:** CloudQuery Plugins\ **Created:** [March 6, 2024, 8:55am UTC](https://community.cloudquery.io/t/issue-with-adding-dependent-tables-in-cloudquery/322 "2024-03-06T08:55:00Z")\ **Posts on this page:** 20\ **Page:** 1
**Author:** ![decent-viper](https://avatars.discourse-cdn.com/v4/letter/d/ecb155/32.png) [@decent-viper](https://community.cloudquery.io/u/decent-viper)\ **Post date:** [March 6, 2024, 8:55am UTC](https://community.cloudquery.io/t/issue-with-adding-dependent-tables-in-cloudquery/322/1 "2024-03-06T08:55:00Z")
I am encountering an issue when I add dependent tables. ---
**Author:** ![herman](https://sea1.discourse-cdn.com/flex001/user_avatar/community.cloudquery.io/herman/32/27_2.png) [@herman](https://community.cloudquery.io/u/herman)\ **Post date:** [March 6, 2024, 8:56am UTC](https://community.cloudquery.io/t/issue-with-adding-dependent-tables-in-cloudquery/322/2 "2024-03-06T08:56:28Z")
Hi, could you please keep your messages in a single thread so that we can keep different conversations separate? Also, would you mind sharing some of the last lines from `cloudquery.log`? ---
**Author:** ![decent-viper](https://avatars.discourse-cdn.com/v4/letter/d/ecb155/32.png) [@decent-viper](https://community.cloudquery.io/u/decent-viper)\ **Post date:** [March 6, 2024, 8:57am UTC](https://community.cloudquery.io/t/issue-with-adding-dependent-tables-in-cloudquery/322/3 "2024-03-06T08:57:29Z")
```auto 2024-03-06T08:54:39Z ERR table resolver finished with error error="GET https://api.github.com/repos/xxxxx/internal-gist/dependency-graph/sbom: 404 Not Found []" client="org: xxxxxxxx repo: xxxxxx/internal-gist" module=github-src table=github_repository_sboms 2024-03-06T08:54:39Z INF table sync finished client="org: xxxx repo: xxxxxx/internal-gist" errors=0 module=github-src resources=1 table=github_repositories 2024-03-06T08:54:39Z INF table sync finished client="org: xxxxx repo: xxxxx/internal-gist" errors=1 module=github-src resources=0 table=github_repository_sboms 2024-03-06T08:54:39Z WRN GitHub secondary rate limit detected for API call: https://api.github.com/repos/xxxx/ops-deployments/dependency-graph/sbom. Sleeping until 2024-03-06T15:00:57+05:30 module=github-src 2024-03-06T08:54:39Z INF table sync finished client=org:xxxxx errors=0 module=github-src resources=7 table=github_installations ``` This is the last few lines from the log: ```auto | Syncing resources... (26/-, 0 resources/hr) [8m37s] ``` Still the same after 8 minutes. ---
**Author:** ![herman](https://sea1.discourse-cdn.com/flex001/user_avatar/community.cloudquery.io/herman/32/27_2.png) [@herman](https://community.cloudquery.io/u/herman)\ **Post date:** [March 6, 2024, 9:04am UTC](https://community.cloudquery.io/t/issue-with-adding-dependent-tables-in-cloudquery/322/4 "2024-03-06T09:04:42Z")
Great, thank you. There seem to be two problems here: - A 404 error for SBOMs. We should raise an issue for this. - The secondary rate limit is hit, causing it to sleep for 30 minutes (?) if I read the timezone correctly. I’ll raise an issue so we can look into this. Do you need the SBOM data, or can you skip it for now? ---
**Author:** ![decent-viper](https://avatars.discourse-cdn.com/v4/letter/d/ecb155/32.png) [@decent-viper](https://community.cloudquery.io/u/decent-viper)\ **Post date:** [March 6, 2024, 9:05am UTC](https://community.cloudquery.io/t/issue-with-adding-dependent-tables-in-cloudquery/322/5 "2024-03-06T09:05:18Z")
I need SBOM data also. Even I need `github_workflows`, but again, same problem. ```auto 2024-03-06T09:07:19Z INF table sync finished client="org: xxxxx repo: xxxxx/secops" errors=0 module=github-src resources=0 table=github_workflows 2024-03-06T09:07:19Z ERR table resolver finished with error error="GET https://api.github.com/repos/xxxxx/internal-gist/dependency-graph/sbom: 404 Not Found []" client="org: xxxxx repo: xxxxx/internal-gist" module=github-src table=github_repository_sboms 2024-03-06T09:07:19Z INF table sync finished client="org: xxxxx repo: xxxx/internal-gist" errors=0 module=github-src resources=1 table=github_repositories 2024-03-06T09:07:19Z INF table sync finished client="org: xxxxx repo: xxxx/internal-gist" errors=1 module=github-src resources=0 table=github_repository_sboms 2024-03-06T09:07:19Z ERR table resolver finished with error error="GET https://api.github.com/repos/xxxxxx/secops/dependency-graph/sbom: 404 Not Found []" client="org: xxxxx repo: xxxxx/secops" module=github-src table=github_repository_sboms 2024-03-06T09:07:19Z INF table sync finished client="org: xxxxx repo: xxxxx/secops" errors=0 module=github-src resources=1 table=github_repositories 2024-03-06T09:07:19Z INF table sync finished client="org: xxxxx repo: xxxxx/secops" errors=1 module=github-src resources=0 table=github_repository_sboms 2024-03-06T09:07:19Z INF table sync finished client="org: xxxxx repo: xxxxx/ops-deployments" errors=0 module=github-src resources=0 table=github_workflows 2024-03-06T09:07:19Z WRN GitHub secondary rate limit detected for API call: https://api.github.com/repos/xxxx/ops-deployments/dependency-graph/sbom. Sleeping until 2024-03-06T15:00:57+05:30 module=github-src 2024-03-06T09:07:19Z INF table sync finished client="org: xxxx repo: xxxxx/internal-gist" errors=0 module=github-src resources=0 table=github_workflows 2024-03-06T09:07:19Z INF table sync finished client=org:xxxx errors=0 module=github-src resources=7 table=github_installations ``` Last few lines after adding `github_workflows`: ```auto / Syncing resources... (24/-, 0 resources/hr) [3m38s] ? ``` What is the issue for `github_workflows`? Any idea how to “solve GitHub secondary rate limit detected for API call”? ---
**Author:** ![herman](https://sea1.discourse-cdn.com/flex001/user_avatar/community.cloudquery.io/herman/32/27_2.png) [@herman](https://community.cloudquery.io/u/herman)\ **Post date:** [March 6, 2024, 9:12am UTC](https://community.cloudquery.io/t/issue-with-adding-dependent-tables-in-cloudquery/322/6 "2024-03-06T09:12:57Z")
I have raised [issue #17040](https://github.com/cloudquery/cloudquery/issues/17040) and [issue #17041](https://github.com/cloudquery/cloudquery/issues/17041) if you’d like to track this. I would recommend removing `sboms` from your table for list for now, and see if the problem persists. Or actually, make sure you add `github_repository_sboms` to your `skip_tables` list; otherwise, it might be included because it’s a child table. ---
**Author:** ![decent-viper](https://avatars.discourse-cdn.com/v4/letter/d/ecb155/32.png) [@decent-viper](https://community.cloudquery.io/u/decent-viper)\ **Post date:** [March 6, 2024, 9:16am UTC](https://community.cloudquery.io/t/issue-with-adding-dependent-tables-in-cloudquery/322/7 "2024-03-06T09:16:52Z")
What about `github_workflows`? ---
**Author:** ![herman](https://sea1.discourse-cdn.com/flex001/user_avatar/community.cloudquery.io/herman/32/27_2.png) [@herman](https://community.cloudquery.io/u/herman)\ **Post date:** [March 6, 2024, 9:17am UTC](https://community.cloudquery.io/t/issue-with-adding-dependent-tables-in-cloudquery/322/8 "2024-03-06T09:17:59Z")
I don’t see any issue with that in the logs, but I think the SBOM table hitting rate limits would cause everything to slow down. ---
**Author:** ![decent-viper](https://avatars.discourse-cdn.com/v4/letter/d/ecb155/32.png) [@decent-viper](https://community.cloudquery.io/u/decent-viper)\ **Post date:** [March 6, 2024, 9:18am UTC](https://community.cloudquery.io/t/issue-with-adding-dependent-tables-in-cloudquery/322/9 "2024-03-06T09:18:09Z")
```auto 2024-03-06T09:19:13Z INF table sync finished client="org: xxxxx repo: xxxxx/ops-deployments" errors=0 module=github-src resources=1 table=github_repositories 2024-03-06T09:19:13Z INF table sync finished client="org: xxxxx repo: xxxxx/operations" errors=0 module=github-src resources=1 table=github_repositories 2024-03-06T09:19:13Z INF table sync finished client="org: xxxxx repo: xxxxx/terraform-templates" errors=0 module=github-src resources=1 table=github_repositories 2024-03-06T09:19:13Z INF table sync finished client="org: xxxxx repo: xxxxx/ops-deployments" errors=0 module=github-src resources=0 table=github_workflows 2024-03-06T09:19:13Z INF table sync finished client="org: xxxxx repo: xxxxx/experimental" errors=0 module=github-src resources=0 table=github_workflows 2024-03-06T09:19:13Z INF table sync finished client="org: xxxxx repo: xxxxx/cloudquery" errors=0 module=github-src resources=0 table=github_workflows 2024-03-06T09:19:13Z INF table sync finished client=org:xxxxx errors=0 module=github-src resources=1 table=github_hooks 2024-03-06T09:19:13Z INF table sync finished client="org: xxxxx repo: xxxxx/terraform-templates" errors=0 module=github-src resources=0 table=github_workflows 2024-03-06T09:19:13Z INF table sync finished client="org: xxxxx repo: xxxxx/internal-gist" errors=0 module=github-src resources=0 table=github_workflows 2024-03-06T09:19:13Z INF table sync finished client=org:xxxxx errors=0 module=github-src resources=7 table=github_installations 2024-03-06T09:19:13Z INF table sync finished client="org: xxxxx repo: xxxxx/secops" errors=0 module=github-src resources=0 table=github_workflows 2024-03-06T09:19:13Z WRN GitHub secondary rate limit detected for API call: https://api.github.com/repos/xxxxx/operations/contents/.github/workflows/console-appserver.yml?ref=main. Sleeping until 2024-03-06T15:00:57+05:30 module=github-src even same for github_workflows? secondary "rate_limits" ? i added sboms in skip tables kind: source spec: # Source spec section name: github path: cloudquery/github registry: cloudquery version: "v8.0.1" tables: [ "github_installations", "github_hooks", "github_repositories", "github_teams", "github_team_members", "github_team_repositories", "github_organizations", "github_organization_members", "github_workflows" ] skip_tables: - github_billing_action - github_billing_package - github_billing_storage - github_external_groups - github_hook_deliveries - github_traffic_clones - github_traffic_paths - github_traffic_referrers - github_traffic_views - github_organization_dependabot_secrets - github_organization_dependabot_alerts - github_releases - github_release_assets - github_repository_dependabot_secrets - github_repository_dependabot_alerts - github_repository_keys - github_workflow_run_usage - github_workflow_jobs - github_repository_branches - github_workflow_runs - github_repository_sboms 2024-03-06T09:24:25Z WRN GitHub secondary rate limit detected for API call: https://api.github.com/orgs/xxxxx/dependabot/secrets. Sleeping until 2024-03-06T15:00:57+05:30 module=github-src same for "github_organization_dependabot_secrets" ``` ---
**Author:** ![herman](https://sea1.discourse-cdn.com/flex001/user_avatar/community.cloudquery.io/herman/32/27_2.png) [@herman](https://community.cloudquery.io/u/herman)\ **Post date:** [March 6, 2024, 9:25am UTC](https://community.cloudquery.io/t/issue-with-adding-dependent-tables-in-cloudquery/322/10 "2024-03-06T09:25:55Z")
Right, thanks, yeah that looks like the same underlying issue. It seems like we will need to make a change to the rate limiting logic. How many repositories do you have in your org? ---
**Author:** ![herman](https://sea1.discourse-cdn.com/flex001/user_avatar/community.cloudquery.io/herman/32/27_2.png) [@herman](https://community.cloudquery.io/u/herman)\ **Post date:** [March 6, 2024, 9:26am UTC](https://community.cloudquery.io/t/issue-with-adding-dependent-tables-in-cloudquery/322/11 "2024-03-06T09:26:51Z")
Can you try setting `concurrency` to a lower value, like maybe `20`? ---
**Author:** ![decent-viper](https://avatars.discourse-cdn.com/v4/letter/d/ecb155/32.png) [@decent-viper](https://community.cloudquery.io/u/decent-viper)\ **Post date:** [March 6, 2024, 9:26am UTC](https://community.cloudquery.io/t/issue-with-adding-dependent-tables-in-cloudquery/322/12 "2024-03-06T09:26:59Z")
Sure. After setting concurrency to a lower value, the rate limit issue is still present. ```auto 2024-03-06T09:27:34Z WRN GitHub secondary rate limit detected for API call: https://api.github.com/repos/xxxx/experimental/releases?per_page=1000. Sleeping until 2024-03-06T15:00:57+05:30 module=github-src ``` ---
**Author:** ![herman](https://sea1.discourse-cdn.com/flex001/user_avatar/community.cloudquery.io/herman/32/27_2.png) [@herman](https://community.cloudquery.io/u/herman)\ **Post date:** [March 6, 2024, 9:31am UTC](https://community.cloudquery.io/t/issue-with-adding-dependent-tables-in-cloudquery/322/13 "2024-03-06T09:31:10Z")
I think this could be because the previous syncs used up the quota, which will reset within an hour. [GitHub Rate Limits Documentation](https://docs.github.com/en/rest/using-the-rest-api/rate-limits-for-the-rest-api?apiVersion=2022-11-28#primary-rate-limit-for-authenticated-users) Our log says it’s a secondary rate limit, but I’m not sure it actually is. It might be the primary one. ---
**Author:** ![decent-viper](https://avatars.discourse-cdn.com/v4/letter/d/ecb155/32.png) [@decent-viper](https://community.cloudquery.io/u/decent-viper)\ **Post date:** [March 6, 2024, 9:31am UTC](https://community.cloudquery.io/t/issue-with-adding-dependent-tables-in-cloudquery/322/14 "2024-03-06T09:31:39Z")
but other tables are working. If the previous quota is used, does that mean it should not work for other tables? For these tables: ```json tables: [ "github_installations", "github_hooks", "github_repositories", "github_teams", "github_team_members", "github_team_repositories", "github_organizations", "github_organization_members", "github_organization_dependabot_alerts", "github_repository_keys" ] ``` It works. ---
**Author:** ![herman](https://sea1.discourse-cdn.com/flex001/user_avatar/community.cloudquery.io/herman/32/27_2.png) [@herman](https://community.cloudquery.io/u/herman)\ **Post date:** [March 6, 2024, 9:32am UTC](https://community.cloudquery.io/t/issue-with-adding-dependent-tables-in-cloudquery/322/15 "2024-03-06T09:32:10Z")
Oh, okay, then yeah it’s probably the secondary rate limit. Can you show your current config? ---
**Author:** ![decent-viper](https://avatars.discourse-cdn.com/v4/letter/d/ecb155/32.png) [@decent-viper](https://community.cloudquery.io/u/decent-viper)\ **Post date:** [March 6, 2024, 9:32am UTC](https://community.cloudquery.io/t/issue-with-adding-dependent-tables-in-cloudquery/322/16 "2024-03-06T09:32:56Z")
```auto kind: source spec: # Source spec section name: github path: cloudquery/github registry: cloudquery version: "v8.0.1" tables: [ "github_installations", "github_hooks", "github_repositories", "github_teams", "github_team_members", "github_team_repositories", "github_organizations", "github_organization_members", "github_organization_dependabot_alerts", "github_repository_keys" ] skip_tables: - github_billing_action - github_billing_package - github_billing_storage - github_external_groups - github_hook_deliveries - github_traffic_clones - github_traffic_paths - github_traffic_referrers - github_traffic_views - github_organization_dependabot_secrets - github_releases - github_release_assets - github_repository_dependabot_secrets - github_repository_dependabot_alerts - github_workflow_run_usage - github_workflow_jobs - github_repository_branches - github_workflow_runs - github_repository_sboms destinations: ["postgresql"] spec: app_auth: - org: cloudquery private_key_path: private-key.pem app_id: "xxxxxx" installation_id: "xxxxxxxx" orgs: ['myorg'] # concurrency: 20 ``` ---
**Author:** ![herman](https://sea1.discourse-cdn.com/flex001/user_avatar/community.cloudquery.io/herman/32/27_2.png) [@herman](https://community.cloudquery.io/u/herman)\ **Post date:** [March 6, 2024, 9:33am UTC](https://community.cloudquery.io/t/issue-with-adding-dependent-tables-in-cloudquery/322/17 "2024-03-06T09:33:58Z")
It seems like the concurrency setting is commented out there? ---
**Author:** ![decent-viper](https://avatars.discourse-cdn.com/v4/letter/d/ecb155/32.png) [@decent-viper](https://community.cloudquery.io/u/decent-viper)\ **Post date:** [March 6, 2024, 9:34am UTC](https://community.cloudquery.io/t/issue-with-adding-dependent-tables-in-cloudquery/322/18 "2024-03-06T09:34:12Z")
Even if I set it the same, there is no improvement. I commented it now. ---
**Author:** ![herman](https://sea1.discourse-cdn.com/flex001/user_avatar/community.cloudquery.io/herman/32/27_2.png) [@herman](https://community.cloudquery.io/u/herman)\ **Post date:** [March 6, 2024, 9:34am UTC](https://community.cloudquery.io/t/issue-with-adding-dependent-tables-in-cloudquery/322/19 "2024-03-06T09:34:54Z")
Can you please share your config when concurrency was set? ---
**Author:** ![decent-viper](https://avatars.discourse-cdn.com/v4/letter/d/ecb155/32.png) [@decent-viper](https://community.cloudquery.io/u/decent-viper)\ **Post date:** [March 6, 2024, 9:35am UTC](https://community.cloudquery.io/t/issue-with-adding-dependent-tables-in-cloudquery/322/20 "2024-03-06T09:35:01Z")
```auto kind: source spec: # Source spec section name: github path: cloudquery/github registry: cloudquery version: "v8.0.1" tables: [ "github_installations", "github_hooks", "github_repositories", "github_teams", "github_team_members", "github_team_repositories", "github_organizations", "github_organization_members", "github_organization_dependabot_alerts", "github_repository_keys", "github_releases" ] skip_tables: - github_billing_action - github_billing_package - github_billing_storage - github_external_groups - github_hook_deliveries - github_traffic_clones - github_traffic_paths - github_traffic_referrers - github_traffic_views - github_organization_dependabot_secrets # - github_organization_dependabot_alerts - github_release_assets - github_repository_dependabot_secrets - github_repository_dependabot_alerts # - github_repository_keys - github_workflow_run_usage - github_workflow_jobs - github_repository_branches - github_workflow_runs - github_repository_sboms destinations: ["postgresql"] spec: app_auth: - org: cloudquery private_key_path: .private-key.pem app_id: "xxxxx" installation_id: "xxxxx" orgs: ['myorg'] concurrency: 20 2024-03-06T09:35:27Z WRN GitHub secondary rate limit detected for API call: https://api.github.com/repos/xxxxx/experimental/releases?per_page=1000. Sleeping until 2024-03-06T16:00:58+05:30 module=github-src ``` [Next page](https://community.cloudquery.io/t/issue-with-adding-dependent-tables-in-cloudquery/322.md?page=2)