Hi team, when we were using Pagerduty plugin v6.2.0 to sync pagerduty_incident_log_entries, we noticed there were some data missing compared to v5.8.0
For a same incident, when we used v5.8.0 to sync log entries, we can get 16 lines of data. But when we used v6.2.0 to sync them, we can only get 14 lines of data. And we didn’t use incremental sync.
We have used Pagerduty API to check, and it should be 16 lines of log entries.
Can someone take a look? Thank you.
Hi @Hide Can you share the configuration spec you’re using (redacted from sensitive information) and CLI version?
Also do you mind looking at the data that you get from version v5.8.0 or the PagerDuty API and see if it has some duplicates based on the ID and CreatedAt fields?
Another thing to check with the data is the earliest CreatedAt and latest CreatedAt fields of the logs entries and of the CreatedAt field of the parent incident. Do you mind sharing those for the missing data?
This is what we used for testing. We need to use log entries with
type = "assign_log_entry"
. But when we used v6.2.0, there were some incidents that didn’t have this type of data. And we checked the log entries using Pagerduty API, there should be assign_log_entry
data.We didn’t check what else log entry data was missing.
Thanks for sharing @Hide, which CLI version are you using? You can check via cloudquery --version
.
With that config pagerduty_incident_log_entries
should be skipped altogether unless you set Source Spec Reference | CloudQuery to true
We are using v5.20.1
FROM ghcr.io/cloudquery/cloudquery:5.20.1 AS builder
I don’t think pagerduty_incident_log_entries is skipped. While some incidents were missing assign log entry data, some other incidents still got those data.
Got it @Hide . Yeah you’re using an old CLI version that’s the reason. We changed the default for skip_dependent_tables
from false
to true
in v6 of the CLI.
To clarify, are you missing both pagerduty_incidents
data and pagerduty_incident_log_entries
data? Or only pagerduty_incident_log_entries
data?
Can you share the following fields from the log entries that’s missing: created_at, id
and the following fields from the incident that has missing log entries: created_at
The PagerDuty plugin has some logic to handle pagination overflow (PagerDuty API is limited to 10000 items per paginated response) and it uses the created_at
to prevent overflowing.
Regardless I recommend updating to the latest CLI ghcr.io/cloudquery/cloudquery:6.14.0
so you have all the recent features and bug fixes.
You’ll need to update your config to set skip_dependent_tables: false
to keep the existing behavior of your configuration
We have updated the CLI and the config file. But the problem still exists.
This is the log entries we were able to get for one incident.
The
created_at
for the missing data is 2025-01-16 09:08:14 UTC
Thanks @Hide that helps!
Can you share the created_at
time of the incident that has these log entries? We start listing log entries using the since
API parameter and use the the incident created_at
for that value
The created_at for that incident is 2025-01-16 09:08:15 UTC.
So is that the problem? But v5.8.0 doesn’t have this issue.
Yeah that’s the issue, when we list log entries we use the incident create_at
as the since
argument for the logs entries API request.
It’s quite surprising there’s a log entry for an incident and that log entry predates the incident
In v5.8.0
we did not have that logic to handle more than 10,000 items, so if you have 10,000 log entries the plugin would error out.
Let check how we can fix this internally
Hi @Hide, can you try with version v6.2.1
of the PagerDuty plugin? That version should fix the issue you’re experiencing
Yeah. It works. Thank you.