Fetching view data in Azure using CQ API payload configuration issues

CloudQuery Plugins
1

Hello folks,

I’m trying to fetch view data in Azure using CloudQuery. I see in the logs that the following API call is being made:

https://management.azure.com/subscriptions/{subscriptionID}/providers/Microsoft.CostManagement/query

The expected columns fetched from the view are very different from the resulting ones. It looks like if the payload of the API is specified in the right way, it returns all the columns as seen in the Azure UI.

Is there a way I can specifically give the payload for the API?

2

Hi, is this question regarding the azure_costmanagement_views or azure_costmanagement_view_queries?

Fetching view queries, we need to convert the date range related parameters (timeframe and daterange) between types, to be able to query data for a saved view. There’s no current way to add additional payload to the query API, but if you have anything in mind, maybe this might be a general feature to be included in the future?

Or did you just want to set arbitrary parameters in the query call?

3

I used both the tables as source in CloudQuery and followed this document to fetch the data: Azure Cost Optimization with CloudQuery.

Is there a way to convert the date range related parameters (timeframe and daterange) between types?

It would be great if we could add the payload. As of now, setting arbitrary parameters would also do. How can I do it?

4

The date range related parameters are converted automatically as the plugin generates the HTTP request to the query endpoint.

What do you mean by payload? The query endpoint takes armcostmanagement.QueryDefinition as payload. Which part of the payload do you want to modify?

I looked at the resource code a little bit more. What the plugin does is it transfers the ReportConfigDefinition (which is the query.* fields under View) into QueryDefinition, and then converts the time range fields (because the time range fields aren’t 100% compatible).

So if you manage to create the View you wish to see in terms of report type etc. in the Azure UI, it should translate automatically to the query results.

5

Created a view with the columns required and desired granularity and group by filter.
Still, I’m able to get only a few columns, whereas the API call to the view gives me a full set of columns that the view has.

API:
https://management.azure.com/subscriptions/<subscription_id>/providers/Microsoft.CostManagement/query?api-version=2023-03-01

Expected Payload that contains the metadata for the above view that gets passed to the above API:

{"type":"ActualCost","dataSet":{"granularity":"Daily","aggregation":{"totalCost":{"name":"Cost","function":"Sum"},"totalCostUSD":{"name":"CostUSD","function":"Sum"}},"sorting":[{"direction":"ascending","name":"UsageDate"}],"grouping":[{"type":"Dimension","name":"ResourceId"},{"type":"Dimension","name":"ResourceType"},{"type":"Dimension","name":"ResourceLocation"},{"type":"Dimension","name":"ChargeType"},{"type":"Dimension","name":"ResourceGroupName"},{"type":"Dimension","name":"PublisherType"},{"type":"Dimension","name":"ServiceName"},{"type":"Dimension","name":"ServiceTier"},{"type":"Dimension","name":"Meter"}]},"timeframe":"Custom","timePeriod":{"from":"2023-09-15T00:00:00+00:00","to":"2023-09-21T23:59:59+00:00"}}
6

I’m opening an issue for this. Can you tell me which version of the Azure plugin you’ve been using? Thanks.
The issue I opened is here

7

Azure: v9.3.4
Thank you so much!
@kemal Is there a way I can track the progress or know till when this might get fixed?

8

You can track the linked issue to see if there’s any progress on it.