CloudQuery GitHub plugin sync error and query on data restriction

CloudQuery Plugins
1

Hi everyone, just discovered CloudQuery yesterday. Looks really cool!

I have a couple of questions about the GitHub plugin.

  1. I gave CloudQuery a newly minted personal access token and configured the spec to fetch info about some private repos. But, when I run cloudquery sync, I am seeing an error:

    cloudquery.yml
....
  tables: ['github_releases']
  destinations: ['file']
  spec:
    access_token: github_pat_<some-token-text>
...

    Error: failed to sync v3 source github: unexpected error from sync client receive: rpc error: code = Unknown desc = failed to sync records: failed to sync unmanaged client: failed to discover repositories: GET https://api.github.com/repos/<my-org>/<my-repo>: 404 Not Found []

    my-org and my-repo are set to the correct values. Any ideas what might be going wrong?

  2. I used a different public GitHub repo and was able to make progress. But here, I wanted to restrict CloudQuery to only download data for the last one month. Is it possible to do that?

2

Hi @next-bluejay :wave:

Would you be able to supply the full configuration file you are using (with any sensitive information redacted) and the CloudQuery CLI version you are using?

Many thanks!

3 
kind: source
spec:
  name: github
  path: cloudquery/github
  registry: cloudquery
  version: "v7.6.4"
  tables: ['github_releases']
  destinations: ['file']
  spec:
    access_token: github_pat_<some-token-text>
    repos: ['my-org/repo1', 'my-org/repo2', 'my-org/repo3']
---
kind: destination
spec:
  name: 'file'
  path: "cloudquery/file"
  registry: "cloudquery"
  version: "v3.4.17"
  spec:
    path: "/tmp/cloudquery/{{TABLE}}/{{UUID}}.{{FORMAT}}"
    format: "json"

I’ve verified that the GitHub PAT is created for my-org and has read-only access to GitHub data (commits, releases, etc.). Any suggestions? I just found that the GitHub PAT created by a user can need an approval before it gets used. Mine was approved just now, and now I don’t see the failed to discover repositories errors.

I still need to figure out how to restrict GitHub data to be downloaded for the last one month.

4

Hi, glad you found the cause of the original error.

Regarding restricting the data to download to the last month, this is not currently possible due to the underlying GitHub SDK not supporting it e.g. list-releases.