Congrats on the progress to open beta on event-based sync for AWS. Just curious how this interplays with the recently added support for GetAccountAuthorizationDetails? The beauty of this API call for AWS is it dramatically reduces the number of API requests required to get all IAM details about an account. This includes users, groups, roles, policies, attachment information, inline policies, permission boundaries, etc.
While it’s great that IAM support is in place as a part of this launch, it looks like it would require enabling the individual IAM tables, which defeats the purpose of GetAccountAuthorizationDetails. While a challenge, it might be something to consider to take the response of GetAccountAuthorizationDetails and break that into the separate table spaces so customers can get the best of both worlds once this capability goes GA.
Hi @mutual-krill - Thank you for that feedback! The GetAccountAuthorizationDetails API is very limited in the data filtering it can provide, which means that any time an IAM resource is updated, the most granular we can get with it is to update all Users, Roles, Groups, LocalManagedPolicies, or AWSManagedPolicies. This is in comparison to the existing solution where the sync will only update the single resource that is updated, so just that singular User, Role, Policy, or Group. In large accounts where the possibility of rate limiting is high, it is important for CloudQuery to use the APIs that will have the largest throughput.
I agree that for batched syncs, the ideal API to use is GetAccountAuthorizationDetails, but for syncing of singular resources, the individual ones are better.
A future enhancement that would unlock the best of both worlds is to generate an SQL view that will combine data from all of the IAM tables to display only the latest/most recent data. This would enable users to sync in bulk the data via GetAccountAuthorizationDetails and then get granular updates to individual resources on an ongoing basis, all while being mindful of rate limits and performance.