I am currently configuring the storage format for the cloudquery.log file. I need the log entries to provide a clear indication of whether the sync process was successful or if there were any errors. I also require the ability to look for specific key-value pairs to retrieve information about the successful sync or the reason behind any errors.
To accomplish this, I’m using the following command:
I have used the command above, and it writes the cloudquery.log in JSON format.
Attached are the logs for successful sync and error sync due to wrong AWS credentials. I want to know the keys that can let me know if it’s a successful sync or a failed sync, and if it failed, why?
Is it this?
{
"Error": "failed to sync v3 source aws": "rpc error": "code = Internal desc = failed to init plugin": "failed to initialize client": "no AWS accounts were successfully configured. See warning messages in the logs for details"
}
There is no single key and message that will tell you the exact error that occurred. Any message that has a level= error will mean that some type of error occurred and something stopped executing. That might be a blocking failure or it might be an ignorable failure, like a single column resolver had an issue (e.g., resource not found).
I would suggest that you generate queries and dashboards for all errors and warnings to begin with and then monitor your system and add appropriate statements to ignore certain errors or warnings that you do not care about. Monitoring a production system is not something that we can boil down to a single message type as there are near infinite failure modes, and each user will care about different things.
We have made an effort to expose the data in ways that users can extract the value that is necessary for their team. If something is missing or if you are seeing an unexpected error, please do let us know.