Is anyone using these successfully when pulling the report generated from the S3 bucket? When doing so get failures when compiling the dbt for the cost_usage_table. I am syncing all of the required tables, but the models do not run for 27 of 34 models.
Hi @jkaser_EA! Are you seeing any specific errors when trying to run the transformation? Any SQL-related issues? Also, did both syncs (S3-CUR and the AWS tables) complete successfully, or were there any errors?
When syncing s3-cur, I have the report coming out as csv, should it not be a csv. This is what the s3-cur sync looks like in the logs.
Starting sync for: s3-cur (cloudquery/s3@v1.3.25) → [postgresql (cloudquery/postgresql@v8.6.3)]
/ Syncing resources… (0/-, 0 resources/hr) [0s]
- Syncing resources… (0/-, 0 resources/hr) [0s]
\ Syncing resources… (0/-, 0 resources/hr) [0s]
| Syncing resources… (0/-, 0 resources/hr) [0s]
/ Syncing resources… (0/-, 0 resources/hr) [0s]
- Syncing resources… (0/-, 0 resources/hr) [0s]
\ Syncing resources… (0/-, 0 resources/hr) [0s]
| Syncing resources… (0/-, 0 resources/hr) [0s]
/ Syncing resources… (0/-, 0 resources/hr) [0s]
-
Syncing resources… (0/-, 0 resources/hr) [1s]
-
Syncing resources… (0/-, 0 resources/hr) [1s]
Needs to be parquet. I will need to update that.
ran the sync now with the correct format of file in the S3 bucket but still getting errors when running the compile for DBT
sync logs:
Starting sync for: s3-cur (cloudquery/s3@v1.3.25) -> [postgresql (cloudquery/postgresql@v8.6.3)]
/ Syncing resources... (0/-, 0 resources/hr) [0s]
- Syncing resources... (0/-, 0 resources/hr) [0s]
\ Syncing resources... (0/-, 0 resources/hr) [0s]
| Syncing resources... (3000/-, 7481 resources/s) [0s]
/ Syncing resources... (3000/-, 5990 resources/s) [0s]
- Syncing resources... (13000/-, 5990 resources/s) [0s]
\ Syncing resources... (13000/-, 5990 resources/s) [0s]
| Syncing resources... (13000/-, 5990 resources/s) [0s]
/ Syncing resources... (13000/-, 5990 resources/s) [0s]
- Syncing resources... (13000/-, 5990 resources/s) [1s]
\ Syncing resources... (13000/-, 11335 resources/s) [1s]
| Syncing resources... (13000/-, 11335 resources/s) [1s]
/ Syncing resources... (13000/-, 11335 resources/s) [1s]
- Syncing resources... (13000/-, 11335 resources/s) [1s]
\ Syncing resources... (13000/-, 11335 resources/s) [1s]
| Syncing resources... (16000/-, 11335 resources/s) [1s]
/ Syncing resources... (17483/-, 10048 resources/s) [1s]
- Syncing resources... (17483/-, 10048 resources/s) [1s]
\ Syncing resources... (17483/-, 10048 resources/s) [1s]
| Syncing resources... (17483/-, 10048 resources/s) [2s]
/ Syncing resources... (17483/-, 10048 resources/s) [2s]
- Syncing resources... (17483/-, 7536 resources/s) [2s]
\ Syncing resources... (17483/-, 7536 resources/s) [2s]
| Syncing resources... (17483/-, 7536 resources/s) [2s]
/ Syncing resources... (17483/-, 7536 resources/s) [2s]
- Syncing resources... (17483/-, 7536 resources/s) [2s]
- Syncing resources... (17483/-, 7536 resources/s) [2s]
Sync completed successfully. Resources: 17483, Errors: 0, Warnings: 0, Time: 3s
dbt compile message:
Starting sync for: s3-cur (cloudquery/s3@v1.3.25) -> [postgresql (cloudquery/postgresql@v8.6.3)]
/ Syncing resources... (0/-, 0 resources/hr) [0s]
- Syncing resources... (0/-, 0 resources/hr) [0s]
\ Syncing resources... (0/-, 0 resources/hr) [0s]
| Syncing resources... (3000/-, 7481 resources/s) [0s]
/ Syncing resources... (3000/-, 5990 resources/s) [0s]
- Syncing resources... (13000/-, 5990 resources/s) [0s]
\ Syncing resources... (13000/-, 5990 resources/s) [0s]
| Syncing resources... (13000/-, 5990 resources/s) [0s]
/ Syncing resources... (13000/-, 5990 resources/s) [0s]
- Syncing resources... (13000/-, 5990 resources/s) [1s]
\ Syncing resources... (13000/-, 11335 resources/s) [1s]
| Syncing resources... (13000/-, 11335 resources/s) [1s]
/ Syncing resources... (13000/-, 11335 resources/s) [1s]
- Syncing resources... (13000/-, 11335 resources/s) [1s]
\ Syncing resources... (13000/-, 11335 resources/s) [1s]
| Syncing resources... (16000/-, 11335 resources/s) [1s]
/ Syncing resources... (17483/-, 10048 resources/s) [1s]
- Syncing resources... (17483/-, 10048 resources/s) [1s]
\ Syncing resources... (17483/-, 10048 resources/s) [1s]
| Syncing resources... (17483/-, 10048 resources/s) [2s]
/ Syncing resources... (17483/-, 10048 resources/s) [2s]
- Syncing resources... (17483/-, 7536 resources/s) [2s]
\ Syncing resources... (17483/-, 7536 resources/s) [2s]
| Syncing resources... (17483/-, 7536 resources/s) [2s]
/ Syncing resources... (17483/-, 7536 resources/s) [2s]
- Syncing resources... (17483/-, 7536 resources/s) [2s]
- Syncing resources... (17483/-, 7536 resources/s) [2s]
Sync completed successfully. Resources: 17483, Errors: 0, Warnings: 0, Time: 3s
The sync completes with out error, but when dbt runs there are errors.
[MainThread]: Done. PASS=5 WARN=0 ERROR=22 SKIP=7 TOTAL=34
On all those with errors this is the message:
relation “cost_usage_table” does not exist
when running the transformation you need to specify the cost and usage report table name as it is saved in your database (usually a combination of its name and billing period)
dbt run --vars '{"cost_usage_table": "<cost_and_usage_report>"}'
for example:
dbt run --vars '{"cost_usage_table": "my_cur_data_BILLING_PERIOD=2024_11"}'
Two different things. Got this error first: Postgres adapter: Postgres error: syntax error at or near “=”
LINE 16: testreports_costreporttest_data_BILLING_PERIOD=2024_11
updated the table name to not have “=” and now has “_”.
After runnig with that table name got a new error:
Postgres adapter: Postgres error: relation “testreports_costreporttest_data_billing_period_2024_11” does not exist
LINE 10: from testreports_costreporttest_data_BILLING_PERIOD_2024_11