Error in CloudQuery custom plugin table resolver execution

Hi, I’ve been able to get a custom plugin using the Python SDK running as a sidecar in K8s. I can see that the tables are successfully getting created in the destination database, but I’m getting this error:

table resolver finished with error

It’s not clear where in my code the problem lies as I can’t print/log/raise exceptions, so the issue stems from this line of the plugin SDK Python. I think it’s either line 106 or 120.

Hi @FireFox :wave:

Is the issue that the log line does not include the exception? What do you see in cloudquery.log?

Yeah, sorry I’ve been debugging in the container.
I modified that error log to include the exception; it’s pointing to something with my code, but yeah, that would be important to include.
Here is the link to the code.
But yeah, +1 for bubbling up exception info wherever you can since there are a lot of abstractions/patterns calling custom code implementations, and if those implementations throw exceptions, devs like myself won’t ever see it.

Yeah, if you want to make a PR, we’d be happy to accept it :slightly_smiling_face: I’m surprised though because I would have thought the exc_info=True would have made it show the exception.

yeah I’ll give it a shot

@herman I’ll put in a PR - exc_info is supposed to be an instance of the exception, not True.
The correct code here would be:

        except Exception as e:
            self._logger.error(
                "table resolver finished with error",
                client_id=client.id(),
                table=resolver.table.name,
                resources=total_resources,
                depth=depth,
                exc_info=e,
            )

The default when it’s set to True is sys.exc_info() which can fail to pick up correctly in multithreaded and async code.
@comic-firefly I’m not sure if you tried this already, but dropping the concurrency down to 1 thread can help significantly when trying to debug these sorts of issues.
GitHub Pull Request #141

Yeah, thanks for this. I was able to debug by modifying the pip installed code. This will be helpful, though. :+1:t2:

Ty!