AWS Plugin - aws_rds_instance_resource_metrics

Hello,
I’m working with the AWS source plugin and the S3 destination plugin in parket format.

I’m using table aws_rds_instances and aws_rds_instance_resource_metrics.
The table of aws_rds_instance_resource_metrics is always empty.

Does there are some config I should pass to indicate i want which metrics ?

The objectif is to get one metric for all database: “Connection” “SUM” “30days”

1 Like

Hi,

  1. Can you share your redacted sync config?
  2. Are there any RDS instances synced for which there are performance insights available?

You can try running fetching the metrics using the AWS CLI (see here) using this command:

aws pi get-resource-metrics \
  --service-type RDS \
  --identifier xxx \
  --start-time xxx \
  --end-time xxx \
  --metric-queries Metric=db.load.avg 

What does the above command return?

Hi,

Sample of source config

kind: source
spec:
  # Source spec section
  name: aws
  path: "cloudquery/aws"
  registry: "cloudquery"
  version: "v27.3.0"
  tables:
    - aws_rds_clusters
    - aws_rds_instance_resource_metrics
    - aws_rds_instances
    - aws_rds_reserved_instances
  destinations: ["s3"]
  spec:
    regions:
      - us-east-1
      - ca-central-1

In the aws_rds_instances table i have multiple entries and nothing in the aws_rds_instance_resource_metrics

I’m not sure about performance insights maybe, I think it’s more Cloudwatch metrics

How to extract one metric for all rds_instances ?

Sample of cli to get metric:

aws-vault exec presentia-prod-legacy -- aws cloudwatch get-metric-statistics \
    --namespace AWS/RDS \
    --metric-name DatabaseConnections \
    --dimensions Name=DBInstanceIdentifier,Value=my-rds-instance\
    --start-time 2023-09-01T00:00:00Z \
    --end-time 2023-10-01T00:00:00Z \
    --period 2592000 \
    --statistics Maximum
{
    "Label": "DatabaseConnections",
    "Datapoints": [
        {
            "Timestamp": "2023-09-01T00:00:00+00:00",
            "Maximum": 16.0,
            "Unit": "Count"
        }
    ]
}

Hi!
We’re currently supporting RDS Performance Insights (represented in the aws_rds_instance_resource_metrics) through Table Options.
CloudQuery will fetch data about RDS Performance Insights that are present there.
You can refer to CQ AWS Table Options documentation
To make things a bit easier, here’s a sample config for RDS Resource metrics:

aws_rds_instances:
  - get_resource_metrics:
      - start_time: "2024-09-01T00:00:00Z"
        end_time: "2024-10-01T00:00:00Z"
        metric_queries:
          - metric: "db.load.min"

Thought I cannot provide the exact metric name which you need, I believe this should help get you started.

1 Like

Hi @JPLemelin,

You should be able to leverage our aws_cloudwatch_metrics and aws_cloudwatch_metric_statistics tables to pull information/data on CloudWatch metrics for RDS instances and other resources.

Make sure to leverage the Table Options documented to specify which statistics to retrieve (e.g. Namespace, Sum, 30-day period). The attributes are similar to those you would specify when using the CLI or when calling the API directly.

@freddy Do you have an exemple ?

You can find an example here. You can use Sum as a value for the statistics attribute.

1 Like

I was able to extrat DatabaseConnections metric, but i got many useless dimentions like ‘EngineName’, ‘DatabaseClass’… I only need ‘DBInstanceIdentifier’

        - list_metrics:
            namespace: AWS/RDS 
            metric_name: DatabaseConnections 
          get_metric_statistics:
            - period: 604800 points.
              start_time: 2024-12-01T00:00:01Z 
              end_time: 2024-12-07T23:59:59Z 
              statistics: [Maximum, Minimum, Sum, Average] 

If the implementation is close to CLI it’s can be more accurate to pass the namespace and metric name to the aws cloudwatch get-metric-statistics instead using aws cloudwatch list-metrics to list metric and then calling get-metric for each

Sample of cli

aws cloudwatch list-metrics --namespace "AWS/RDS" --metric-name "DatabaseConnections"
{
    "Metrics": [
        {
            "Namespace": "AWS/RDS",
            "MetricName": "DatabaseConnections",
            "Dimensions": [
                {
                    "Name": "DBInstanceIdentifier",
                    "Value": "my-db"
                }
            ]
        },
        {
            "Namespace": "AWS/RDS",
            "MetricName": "DatabaseConnections",
            "Dimensions": [
                {
                    "Name": "EngineName",
                    "Value": "postgres"
                }
            ]
        },
        {
            "Namespace": "AWS/RDS",
            "MetricName": "DatabaseConnections",
            "Dimensions": []
        },
        {
            "Namespace": "AWS/RDS",
            "MetricName": "DatabaseConnections",
            "Dimensions": [
                {
                    "Name": "DatabaseClass",
                    "Value": "db.t4g.medium"
                }
            ]
        }
    ]
}
aws cloudwatch get-metric-statistics \
    --namespace AWS/RDS \
    --metric-name DatabaseConnections \
    --start-time 2024-12-01T00:00:00Z \
    --end-time 2024-12-07T23:59:59Z \
    --period 604800 \
    --statistics Maximum Minimum Average Sum
{
    "Label": "DatabaseConnections",
    "Datapoints": [
        {
            "Timestamp": "2024-12-01T00:00:00+00:00",
            "Average": 17.524305555555557,
            "Sum": 176645.0,
            "Minimum": 0.0,
            "Maximum": 19.0,
            "Unit": "Count"
        }
    ]
}

I try to pass namespace and metric-name directly to to the get_metric_statistics table options like this

        - get_metric_statistics:
            - namespace: AWS/RDS 
              metric_name: DatabaseConnections
              period: 604800 
              start_time: 2024-12-01T00:00:01Z 
              end_time: 2024-12-07T23:59:59Z 
              statistics: [Maximum, Minimum, Sum, Average] 

But i got this error

Error: failed to sync v3 source aws: failed to init source aws: rpc error: code = Internal desc = failed to init plugin: failed to initialize client: invalid table_options: invalid input: cannot set Namespace in CloudwatchMetrics.GetMetricStatisticsOpts

Any hint will be appreciate
JP

Hi @JPLemelin,

namespace, dimensions, and metric_name are not available to be set on get_metric_statistics. Instead they are gathered from each result from list_metrics.

I see that previously you had tried that method but ended up with many extraneous dimensions.

The following includes a dimension filter only for DBInstanceIdentifier. Does this work for you?

        - list_metrics:
            namespace: AWS/RDS
            metric_name: DatabaseConnections
            dimensions:
              - name: DBInstanceIdentifier
          get_metric_statistics:
            - period: 604800
              start_time: 2024-12-01T00:00:01Z
              end_time: 2024-12-07T23:59:59Z
              statistics: [Maximum, Minimum, Sum, Average]
1 Like

Got it! :smiley:

I did not try it since the CLI require dimension to be a Key Pair like --dimensions "Name=DBInstanceIdentifier,Value=my-db"; value is required and can not be null.

So, thanks for your help @freddy and @Riley_Wilburn !!