# Unable to run cloudquery syncs in parallel seeking configuration guidance

**URL:** <https://community.cloudquery.io/t/unable-to-run-cloudquery-syncs-in-parallel-seeking-configuration-guidance/235>\
**Category:** CloudQuery Plugins\
**Created:** [April 24, 2024, 11:18pm UTC](https://community.cloudquery.io/t/unable-to-run-cloudquery-syncs-in-parallel-seeking-configuration-guidance/235 "2024-04-24T23:18:13Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![selected-bonefish](https://avatars.discourse-cdn.com/v4/letter/s/ecd19e/32.png) [@selected-bonefish](https://community.cloudquery.io/u/selected-bonefish)\
**Post date:** [April 24, 2024, 11:18pm UTC](https://community.cloudquery.io/t/unable-to-run-cloudquery-syncs-in-parallel-seeking-configuration-guidance/235/1 "2024-04-24T23:18:13Z")

</div>

Hi CQ folks,

I am trying to run CloudQuery syncs in parallel, but it seems like the execution happens sequentially. Just curious if I am missing any setting in my configuration below that should be used to run the syncs from different sources in parallel.

```yaml
kind: source
spec:
  name: aws-us-west-2
  registry: local
  path: /app/plugins/aws
  tables:
    - aws_*
  destinations: ["postgresql"]
  spec:
    concurrency: 100
    initialization_concurrency: 4
    aws_debug: false
    regions:
      - "us-west-2"
    org:
      member_role_name: ${MEMBER_ROLE_NAME}
      admin_account:
        id: dev
        role_arn: ${ASSUME_ROLE_ARN}
---
kind: source
spec:
  name: aws-ap-northeast-1
  registry: local
  path: /app/plugins/aws
  tables:
    - aws_*
  destinations: ["postgresql"]
  spec:
    concurrency: 100
    initialization_concurrency: 4
    aws_debug: false
    regions:
      - "ap-northeast-1"
    org:
      member_role_name: ${MEMBER_ROLE_NAME}
      admin_account:
        id: dev
        role_arn: ${ASSUME_ROLE_ARN}
---
kind: destination
spec:
  name: postgresql
  registry: local
  path: /app/plugins/postgresql
  write_mode: append
  spec:
    connection_string: ${PG_CONNECTION_STR}

```

---

<div class="post-metadata">

**Author:** ![ben](https://sea1.discourse-cdn.com/flex001/user_avatar/community.cloudquery.io/ben/32/88_2.png) [@ben](https://community.cloudquery.io/u/ben)\
**Post date:** [April 24, 2024, 11:22pm UTC](https://community.cloudquery.io/t/unable-to-run-cloudquery-syncs-in-parallel-seeking-configuration-guidance/235/2 "2024-04-24T23:22:30Z")

</div>

Hi @selected-bonefish,

You are correct. All `source` blocks in a single sync will be executed sequentially. If you are interested in running syncs in parallel, you should put each `source` in its own container and run the containers in parallel.

It looks like you are using the AWS plugin, so I would suggest checking out the ECS Deployment guide to run parallel ECS tasks: [ECS Deployment Guide](https://docs.cloudquery.io/docs/deployment/ecs).

Alternatively, you can run jobs in parallel on EKS or any Kubernetes cluster: [Kubernetes Deployment Guide](https://docs.cloudquery.io/docs/deployment/kubernetes).

---

<div class="post-metadata">

**Author:** ![selected-bonefish](https://avatars.discourse-cdn.com/v4/letter/s/ecd19e/32.png) [@selected-bonefish](https://community.cloudquery.io/u/selected-bonefish)\
**Post date:** [April 24, 2024, 11:27pm UTC](https://community.cloudquery.io/t/unable-to-run-cloudquery-syncs-in-parallel-seeking-configuration-guidance/235/3 "2024-04-24T23:27:26Z")

</div>

I see, thanks for the clarification. Not sure if it’s just me, but it seems like the doc ([Running CloudQuery in Parallel](https://docs.cloudquery.io/docs/advanced-topics/running-cloudquery-in-parallel)) is kind of misleading. I will move each source to a different container. Thanks!

---

<div class="post-metadata">

**Author:** ![ben](https://sea1.discourse-cdn.com/flex001/user_avatar/community.cloudquery.io/ben/32/88_2.png) [@ben](https://community.cloudquery.io/u/ben)\
**Post date:** [April 25, 2024, 11:37am UTC](https://community.cloudquery.io/t/unable-to-run-cloudquery-syncs-in-parallel-seeking-configuration-guidance/235/4 "2024-04-25T11:37:03Z")

</div>

Thanks for that feedback! If you can point me to the piece that tripped you up, we will rework it to be less ambiguous.

---

<div class="post-metadata">

**Author:** ![alfredgamulo](https://sea1.discourse-cdn.com/flex001/user_avatar/community.cloudquery.io/alfredgamulo/32/51_2.png) [@alfredgamulo](https://community.cloudquery.io/u/alfredgamulo)\
**Post date:** [November 19, 2024, 6:56pm UTC](https://community.cloudquery.io/t/unable-to-run-cloudquery-syncs-in-parallel-seeking-configuration-guidance/235/5 "2024-11-19T18:56:24Z")

</div>

The documentation at [Running CloudQuery in Parallel | CloudQuery](https://docs.cloudquery.io/docs/advanced-topics/running-cloudquery-in-parallel#splitting-syncs-manually) implies that a config file with multiple source blocks will run in parallel.

> When splitting a sync into multiple source-integration configurations to be run in parallel

Secondly, if a config file has multiple sources, how does it affect the “sharding” feature?