Jira Plugin Configuration: add MaxResults as configuration option

CloudQuery Plugins
1

I was testing the jira plugin on my internal jira server and it failed collecting the issues due to timeout.

Would it be possible making the MaxResults variable as a configuration parameter ? By default it is 1000 but being able to change to 100, for example, would work better.

Thank you.

Regards

2

Hey @Alex_Scal :waving_hand: Thank you for opening this!

The issues endpoint is paginated, and the MaxResults parameter is controlling the maximum size of each page. Looks like the hardcoded value of 1000 is causing the timeout you’re seeing.

I wonder if we should:

  • Have a configuration for it, as you suggested.
  • Have a better hardcoded default, because 1000 is just too much (default is 50, which we didn’t use because pagination was too slow).
  • Adjust dynamically upon seeing those timeouts, rather than adding a configuration.

Could you please share the exact error you’re seeing? Let’s try to make sure this is indeed what’s going on.

3

hi @cq_mariano
thanks for your prompt reply!

This is what I saw in the logs

{“level”:“error”,“client”:“jira”,“error”:“No response returned: Get “https://jira.xxx.yyy/rest/api/2/search?expand=renderedFields%2Cnames%2Cschema%2Ctransitions%2Coperations%2Ceditmeta%2Cchangelog%2Cfields&fields=*all&maxResults=1000\”: GET https://jira.xxx.yyy/rest/api/2/search?expand=renderedFields%2Cnames%2Cschema%2Ctransitions%2Coperations%2Ceditmeta%2Cchangelog%2Cfields&fields=*all&maxResults=1000 giving up after 5 attempt(s)”,“invocation_id”:“xxxxx”,“message”:“table resolver finished with error”,“module”:“jira-source”,“table”:“jira_issues”,“time”:“xxx”,“time”:“xxx”}
{“level”:“info”,“client”:“jira”,“duration_ms”:615110.974568,“errors”:1,“invocation_id”:“xxx”,“message”:“table sync finished”,“module”:“jira-source”,“resources”:0,“table”:“jira_issues”,“time”:“xxx”,“time”:“xxx”}

I have tried the same URL with postman and I got the same timeout (which is due to my own server not able to handle that amount of data in the response)

My suggestion would be to have something like 500 as default and have the optional parameter available in the config. It is not worth it, IMHO, to change the code to add a dynamic changing value at runtime.

That suggestion would be better suited for the GitHub plugin which hits the rate limit very quickly.

Thank you!

4

Alex, I created an internal issue to look into this. It’s currently in our backlog and will be triaged by our team.

Just to double-check, since you’ve tried this on Postman, does it work when you go down to 500?

5

hi @cq_mariano

it works with 500. So 500 could be left as default and let the user to override the MaxResults if needed.

Thank you.