I was PoCing sending data from the Facebook Ads (source) plugin to PostgreSQL (destination) and encountered two roadblocks:
I had no luck finding data such as CTR, clicks, impressions, or reach in the dedicated schemas. Do we have a schema that includes these fields?
After syncing ~800 records from the facebookmarketing_ads table/schema, we received an error message that is not really informative (“unknown error”). Did anyone encounter such an error? It doesn’t look like reaching the API limit. Partial logs attached in the thread.
2023-11-03T13:47:30Z INF Start fetching resources destinations=["postgresql (cloudquery@cloudquery/postgresql)"] module=cli source="facebookmarketing (cloudquery@cloudquery/facebookmarketing)"
2023-11-03T13:47:30Z INF started call grpc.component=server grpc.method=Sync grpc.method_type=server_stream grpc.service=cloudquery.plugin.v3.Plugin grpc.start_time=2023-11-03T14:47:30+01:00 grpc.time_ms=0.097 module=cli peer.address= protocol=grpc
2023-11-03T13:47:30Z INF top level table resolver started client=[REMOVED BEFORE SHARING LOGS] module=facebookmarketing-src table=facebookmarketing_ads
2023-11-03T13:47:30Z INF started call grpc.component=server grpc.method=Write grpc.method_type=client_stream grpc.service=cloudquery.plugin.v3.Plugin grpc.start_time=2023-11-03T14:47:30+01:00 grpc.time_ms=6.605 module=cli peer.address= protocol=grpc
2023-11-03T13:47:36Z INF Migrating table module=pg-dest table=facebookmarketing_ads
2023-11-03T13:47:36Z INF Table exists, auto-migrating module=pg-dest table=facebookmarketing_ads
2023-11-03T13:50:13Z ERR table resolver finished with error error="http status 500: (1) : An unknown error occurred" client=[REMOVED BEFORE SHARING LOGS] module=facebookmarketing-src table=facebookmarketing_ads
About the second part of your question, that looks like an error from the Facebook servers (status code 500). I’m afraid there isn’t much we can do to fix that; you will need to reach out to the Facebook team about that one.
It might also have been a temporary issue on their side. Maybe retrying now will help.
For the first part of your question, I’m not sure, but looking at the documentation, it seems like tracking_specs and conversion_specs on the ads table (Facebook Ads Documentation) might have some of the information.
If not, could you check out this API and see if it would get what you need? Marketing API Insights
It looks like this includes impressions and clicks. If it looks right to you, we can open a GitHub issue to get this added to the source plugin.
Regarding point 1: Yes, the Marketing Insights API is what marketers and data professionals usually use when working on reports and statistical models around “attribution” questions and problems. The tracking_specs and conversion_specs do not seem to include what is typically needed.
I would love to see this being included. If we decide to open a GitHub issue, I recommend adding a note to ensure we can use “breakdowns” with the Insights API call (Insights Breakdowns Documentation).
Regarding point 2: It seems like it. Is there any way to control API throttling (e.g., limiting the number of requests per hour or batching the requests) that is implemented?
Thanks! That’s useful. Would you mind opening the GitHub issue with any details about what you’d like to have included?
As for limiting concurrent requests: you can use the concurrency setting. It defaults to 1000, but if you lower it to, say, 4, then it will drastically limit the number of concurrent requests.
Actually, I’ll open a GitHub issue as a placeholder and link it here shortly, but feel free to add to that if you’d like to add more details.
