Help with logging and skipping errors in CloudQuery source plugin

musical-ray · January 4, 2024, 9:48pm

Hi folks,

New user of CloudQuery here, and we wrote a source plugin using the Go SDK. I need to skip and log certain errors from the API we are querying. Is there any example of logging from a plugin available?

ben · January 4, 2024, 9:49pm

Hi @musical-ray - that is great to hear! Let me find an example for you.

ben · January 4, 2024, 9:54pm

Here is an example:

github.com

cloudquery/cloudquery/blob/5b95470e248c3e5add3fd41aa2aba788d754387c/plugins/source/aws/resources/services/rds/db_parameter_groups.go#L72-L79


      
          		if err != nil {
          			if client.IsAWSError(err, "DBParameterGroupNotFound") {
          				cl.Logger().Warn().Err(err).Msg("received DBParameterGroupNotFound on DescribeDBParameters")
          				return nil
          			}
          			return err
          		}
          		res <- page.Parameters

In this case, if the API returns a certain error, then we will log a warning (but it could be an Error()) and return without an error.

The thing that is important to do is add the logger to the plugin Client so that all of the resources can access it during the sync:

github.com

cloudquery/cloudquery/blob/5b95470e248c3e5add3fd41aa2aba788d754387c/plugins/source/aws/client/client.go#L73-L75


      
          func (c *Client) Logger() *zerolog.Logger {
          	return &c.logger
          }

musical-ray · January 4, 2024, 9:58pm

tysm! Will try it now

ben · January 4, 2024, 9:58pm

Great! Let me know if you have any questions or run into any issues!

musical-ray · January 5, 2024, 5:01am

Worked perfectly. Thanks again!

Topic		Replies	Views
Cloudquery source error handling and logging issues CloudQuery Plugins	4	4	April 23, 2024
CloudQuery logs proxy connection errors incorrectly formatted and repeated CloudQuery Plugins	2	1	October 24, 2023
Need help with cloudquery log format for clear sync process indicators CloudQuery Plugins	2	4	November 7, 2023
Error in CloudQuery custom plugin table resolver execution CloudQuery Plugins	6	1	March 8, 2024
Error retrieving plugin URL for CloudQuery Postgres to BigQuery transfer CloudQuery Plugins	11	2	November 8, 2023

Help with logging and skipping errors in CloudQuery source plugin

Related topics