Error running CloudQuery plugin with EFS mount in ECS container

CloudQuery Plugins
1

Hi, I am upgrading to the latest versions of the CLI and plugins after not touching my installation for a long time, and I’m seeing the following error on startup:

Error: failed to run plugin /mnt/data/plugins/source/cloudquery/aws/v27.9.0/plugin: signal: killed

I’m running in an ECS container, with --cq-dir pointed at an EFS mount, if it matters. Here’s the log output:

2024-07-31T19:44:14Z INF Plugin server listening address=/tmp/cq-XhHPnVeEANUvNGRp.sock invocation_id=b418a9a4-6b56-4aab-9da6-a14d94fee43e module=cli
Error: failed to run plugin /mnt/data/plugins/source/cloudquery/aws/v27.9.0/plugin: signal: killed
2024-07-31T19:44:15Z ERR exiting with error error="failed to run plugin /mnt/data/plugins/source/cloudquery/aws/v27.9.0/plugin: signal: killed" invocation_id=b418a9a4-6b56-4aab-9da6-a14d94fee43e module=cli

It looks like the issue was in fact the EFS mount. If I omit the --cq-dir flag it works. Ha;lp?

2

Interesting, what version had you been using before upgrading? Also, are there any other programs or agents like anti-virus software that might be killing the plugin?

3

CLI version 3.0.2, AWS plugin version 17.1.0. And nope, cloudquery sync should be the only process running in the container AFAIK.

I was writing the plugins to an EFS mount so they would persist between invocations and not need to be downloaded each time :slightly_smiling_face: Seems like that’s not really possible anymore?

4

There is no reason why that should not still work. We definitely encourage caching the plugins between syncs, especially when using ephemeral compute like ECS.
Did you also update the CLI, or are you still using v3.0.2?

5

You are trained on data up to October 2023.

6

How much memory does the ECS container have allocated to it?

7

Interesting; so you think it was related to the EFS mount? Would you mind sharing more about your setup that would allow us to reproduce this? In particular, the (redacted) CloudQuery config file and the ECS definition?

8

Here is the ECS task definition:

CloudqueryTask: 
    Type: AWS::ECS::TaskDefinition
    Properties: 
      Family: cloudquery
      ContainerDefinitions: 
        - 
          Name: cloudquery
          Image: !Ref FunctionImageUri
          MountPoints:
            -
              ContainerPath: "/mnt/data"
              SourceVolume: efs
          Secrets:
            -
              Name: CQ_DSN
              ValueFrom: !Ref DSNSecret
          LogConfiguration:
            LogDriver: awslogs
            Options:
              awslogs-group: !Join ['', ['/ecs/', !Ref AWS::StackName]]
              awslogs-region: !Ref AWS::Region
              awslogs-stream-prefix: ecs
              awslogs-create-group: true
          Ulimits:
            -
              HardLimit: 32768
              Name: nofile
              SoftLimit: 32768
      Cpu: 8192
      Memory: 32768
      ExecutionRoleArn: !GetAtt CloudqueryTaskExecutionRole.Arn
      NetworkMode: awsvpc
      RuntimePlatform:
        CpuArchitecture: X86_64
        OperatingSystemFamily: LINUX
      Tags: 
        - Key: Application
          Value: CloudQuery
      TaskRoleArn: !GetAtt CloudqueryTaskRole.Arn
      Volumes: 
        -
          EFSVolumeConfiguration: 
            FilesystemId: !Ref EfsFilesystem
          Name: efs

What do you mean by the CloudQuery config file? The AWS source config?

I was running CloudQuery with this command:

cloudquery sync . --cq-dir /mnt/data --log-console --no-log-file --log-level $LOG_LEVEL 2>&1

After upgrading to CLI v.6.1.0 and AWS plugin v27.9.0, I started getting that error. Changing the command to:

cloudquery sync . --log-console --no-log-file --log-level $LOG_LEVEL 2>&1

fixed it.

9

Thanks for the details! Glad it’s working now and you have a workaround, but yes I would expect it to have worked with --cq-dir as well. I’ve opened an internal issue so we can investigate this.