CloudQuery heartbeat request and logging level behavior clarification

CloudQuery Plugins
1

Hi, I had requested a way to get a heartbeat from a running process here. The proposed solution was to use --tables-metrics-location. However, I think that this is a local file and would not be accessible to a user running the system in a containerized batch process.

I kind of wish the info logging didn’t post resource details and that level of information was for debug instead. Is that still the behavior or have things improved with respect to that recommendation?

2

Hello, the first proposed solution you listed was

This could be addressed by a log that indicates which table is being collected by the source module.

Where would you like this log to be? Would it also not be local just like the --tables-metrics-location solution?

3

I guess I meant a log in the same output stream.

4

However, I think that this is a local file and would not be accessible to a user running the system in a containerized batch process.

Hi @neat-kingfish, I think there’s a solution to mount a volume and write the metrics file to the volume.

I kind of wish the info logging didn’t post resource details and that level of information was for debug instead.

Is that related to the heartbeat request? Can you also share the exact log message (an example would be good) you’re referring to?

5

Here’s an example info log that I’m saying should be at the debug level:

{
    "level": "info",
    "module": "pgx",
    "invocation_id": "76b776ea-6db4-4d6d-a054-0e5e28dc753e",
    "args": [
        "2024-08-28T17:34:22.859235Z",
        "aws-<redacted>",
        "0626df0d-954c-5415-b9af-81d5d4226f14",
        null,
        "<redacted>",
        "ANPAZKAPJZG4ATKMS5NQP",
        null,
        "arn:aws:iam::aws:policy/<redacted>",
        0,
        "2024-07-27T00:35:42Z",
        "v1",
        null,
        true,
        "/",
        0,
        "<redacted>",
        "Amazon<redacted>RolePolicy",
        "2024-07-27T00:35:42Z",
        "<redacted> (truncated 40 bytes)",
        "<redacted>"
    ],
    "commandTag": "INSERT 0 1",
    "message": "BatchQuery",
    "sql": "insert into \"aws_iam_policies\" (\"_cq_sync_time\",\"_cq_source_name\",\"_cq_id\",\"_cq_parent_id\",\"account_id\",\"id\",\"tags\",\"arn\",\"attachment_count\",\"create_date\",\"default_version_id\",\"description\",\"is_attachable\",\"path\",<redacted>\"input_hash\"=excluded.\"input_hash\"",
    "time": "2024-08-28T17:39:15Z"
}

Where info could just be the table name being accessed.

6

Hi, I’ll discuss this with the team and will get back to you.