We do not have a way to check that built it; it mostly depends on how you are running the sync. It usually tends to fill all the memory available on the system.
Are you running it locally using the CLI, or are you using Kubernetes/Docker containers, a VM, etc?
I think those are based on t2.micros with a limit of 1GB of memory and no swap, that’s why awspricing is crashing as it’s hitting a hard memory limit.
We are already tracking an issue for this and have recently started looking into possible optimizations, but as it is right now, the plugin won’t work on that setup as it requires additional memory. #15017
What you can do is try running on an instance with more memory and ideally swap enabled (so if it still bursts the limit, swap will take the weight off). Let us know if you’re able to, we’d love to get some feedback out of that. What we will do is prioritize the investigation of these improvements for awspricing. You can also subscribe for notifications on the issue above.
@stefan, is this an issue in every version of the awspricing plugin?
If not, can you let me know which version is better so that we won’t run into this issue?
Yeah, I’ve reduced the services. But I’ve some doubts in the data we got.
@stefan, I am running this awspricing for one account for RDS, ElasticCache, and EC2 services in the us-east-1 region. How can I check the costs that are incurred for RDS, as well as the network performance and all?
Can you help me understand how to check this data and the total pricing for RDS or individual pricing for each RDS instance?
It should give you a place to start exploring the data. If you have any questions after that, let me know. I can help with queries to fetch specific data that interests you!
I’ve read these docs and now I understand the relationship between these two tables. As suggested in the above docs, can we use the command below to create a new table in CloudQuery directly?
WITH expanded_price_dimensions AS (
SELECT
st.sku,
st.type,
st.effective_date,
jsonb_array_elements(st.price_dimensions) AS price_dimension
FROM
awspricing_service_terms AS st
)
SELECT
sp.sku,
sp.product_family,
sp.attributes->>'regionCode' AS region,
sp.attributes->>'instanceType' AS instance_type,
epd.effective_date,
epd.price_dimension->'pricePerUnit'->>'USD' AS price_per_unit_usd,
epd.price_dimension->>'description' AS price_description,
epd.price_dimension->>'unit' AS unit
FROM
awspricing_service_products AS sp
JOIN
expanded_price_dimensions AS epd
ON
sp.sku = epd.sku
WHERE
sp.attributes->>'servicecode' = 'AmazonEC2'
AND epd.type = 'OnDemand';
Like, can the output of this command be stored in another table named pricing_options?
In this specific example, you’re searching for EC2 instance pricing, which makes sense for this service type, but would not make sense for the other ones.
We currently do not have a transformation profile for the AWS Pricing plugin, but feel free to create a feature request for that on our CloudQuery repository and we could consider implementing it. CloudQuery Repository
Alternatively, you could:
Create your own transformation from CloudQuery Cloud, in the Developers > Addons section, or even
Manually create a SQL view for that specific use case.
