Issue adding tables to crowdstrike plugin context cancelled error

Hey all! I’m trying to add more tables to the CrowdStrike plugin and saw <@726919581708451920> has a PR to upgrade it to use v4 SDK. I tried adding a new table off of his branch, but I’m getting a ‘context canceled’ error:

2023-12-19T07:08:08Z ERR table resolver finished with error error="could not query hosts: Get \"https://api.us-2.crowdstrike.com/devices/queries/devices/v1?limit=100&offset=0\": Post \"https://api.us-2.crowdstrike.com/oauth2/token\": context canceled" client=CrowdStrike module=crowdstrike-src table=crowdstrike_falcon_zta

Also happens on any table I put into the config file. Any pointers on where I can start to pinpoint the root cause?

Hi @splendid-moray, can you share the CLI version you’re using and configuration spec? Best if you always use the latest CLI (now v4.3.4) and plugins versions.

Hey all! I’m trying to add more tables to the CrowdStrike plugin. Is this the PR https://github.com/justmiles/cq-source-crowdstrike/pull/7 in question? I’ll try to run it too.

Looks like the issue is the Falcon client using context from Configure.
@splendid-moray pushed a new commit in the PR, let me know if it works for you.

It works now! Thanks @kemal. What’s the difference between creating the Falcon client from Configure vs Sync?

Glad to hear :slightly_smiling_face: Creating it in Configure() it was also using the context.Context from that, but that context is not long-lived (and gets cleaned up after Configure returns).

So when we create the client and persist it as services in the Client struct, we kind of end up with a client using an already-cancelled context.

Ah, that makes sense. So in short, use Configure to configure CloudQuery and Sync to configure/setup the actual API client.

P.S. I created a PR: https://github.com/disq/cq-source-crowdstrike/pull/1 @kemal. Not sure if I should create a PR on @light-iguana’s repo or yours.

I think it should be on justmiles, but justmiles should first merge my PR. What you can do is open it on justmiles’ repo but base it off my PR (so, both can get merged, in order).

Also, if there’s an error, you shouldn’t fmt.Print it; either return the err or use c.Logger and log it:

c.Logger.Warn().Err(err).Msg("could not get ZTA")

The resource can benefit from a “PreResourceResolver”. That’s when we fetch a list of things and pass it into the channel first, then handle them in another e.g. “getZTA” function and get details and override resource.Item.

Lots of examples can be found in the main repo. So you would first do:

res <- queryResponse.Resources

Then in the getZTA function, look each one up.

Something like this?

Hmm, how would this work though with the PreResourceResolver function? Since one host can have many zero trust assessments, can I just return the list back by doing resource.Item = response.Resources? (response.Resources is a list of pointers to structs)

Yes, something like you linked above. In the example above, response.Steps is a slice. The SDK “unwraps” that slice and getStep is called once for each element. As long as getStep doesn’t produce more than one element, it’s OK to just overwrite response.Item.

However, as you mentioned, if there’s a possibility of producing more than one element in the PreResourceResolver, then this pattern doesn’t work, and it has to be done in the main resolver (like you do in the PR).

Looking at your PR again, it looks like ztaResponse.Resources is plural, so my suggestion won’t work.

I’m going to try to use Relations on the Hosts table. I think this would work (still testing it).

GitHub Pull Request

I did it differently and don’t think I understand relations enough to effectively use it.

Thank you for supporting this effort! I truly appreciate it. Really pleasant DX too!

Just merged these. Thanks, guys!