# CloudQuery S3 destination JSON fields landing as base64-encoded is it configurable

**URL:** <https://community.cloudquery.io/t/cloudquery-s3-destination-json-fields-landing-as-base64-encoded-is-it-configurable/386>\
**Category:** CloudQuery Plugins\
**Created:** [February 9, 2024, 9:04pm UTC](https://community.cloudquery.io/t/cloudquery-s3-destination-json-fields-landing-as-base64-encoded-is-it-configurable/386 "2024-02-09T21:04:21Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![pure-quetzal](https://avatars.discourse-cdn.com/v4/letter/p/ac91a4/32.png) [@pure-quetzal](https://community.cloudquery.io/u/pure-quetzal)\
**Post date:** [February 9, 2024, 9:04pm UTC](https://community.cloudquery.io/t/cloudquery-s3-destination-json-fields-landing-as-base64-encoded-is-it-configurable/386/1 "2024-02-09T21:04:21Z")

</div>

Hello folks! We are using the S3 destination, writing in Parquet format. We have a number of JSON fields across our data sources. They seem to be landing in S3 as base64-encoded.

Can you please confirm this behavior? Is it configurable? Can you point me to the source code?

Many questions 🙂 Thank you!

---

<div class="post-metadata">

**Author:** ![yevgenyp](https://avatars.discourse-cdn.com/v4/letter/y/3be4f8/32.png) [@yevgenyp](https://community.cloudquery.io/u/yevgenyp)\
**Post date:** [February 10, 2024, 2:27pm UTC](https://community.cloudquery.io/t/cloudquery-s3-destination-json-fields-landing-as-base64-encoded-is-it-configurable/386/2 "2024-02-10T14:27:24Z")

</div>

Hey 👋,

The docs for the S3 plugins are here: [S3 Plugin Documentation](https://hub.cloudquery.io/plugins/destination/cloudquery/s3/v4.8.9/docs) and the source is here: [S3 Plugin Source](https://github.com/cloudquery/cloudquery/tree/main/plugins/destination/s3).

What warehouse do you use to query it? Is it Athena? If yes, there is a flag ([S3 Overview](https://hub.cloudquery.io/plugins/destination/cloudquery/s3/v4.8.9/docs?search=s3#overview-s3-spec)) that you need to use called `athena` that should make JSON columns compatible with Athena as different destinations expect the JSON columns in slightly different ways.

I’ll also let other folks weigh in here on Monday in case I’m not up to date with the S3 destination.

---

<div class="post-metadata">

**Author:** ![herman](https://sea1.discourse-cdn.com/flex001/user_avatar/community.cloudquery.io/herman/32/27_2.png) [@herman](https://community.cloudquery.io/u/herman)\
**Post date:** [February 12, 2024, 10:56am UTC](https://community.cloudquery.io/t/cloudquery-s3-destination-json-fields-landing-as-base64-encoded-is-it-configurable/386/3 "2024-02-12T10:56:15Z")

</div>

Hi 👋

So the CloudQuery JSON type is an Arrow extension, which uses binary as its base type. I can imagine how a bug there, or maybe in the Arrow parquet writer when dealing with extension types, could lead to it being stored as base64, but it shouldn’t be the case as far as I know.

I also just tested it myself, loading a parquet file with JSON columns from S3 and it showed as `varchar`, not base64 encoded. I tested by downloading the file and then importing it with DuckDB.

Could you share a reproduction by any chance? Maybe the source plugin is using an older version of the SDK, or a different language?

---

<div class="post-metadata">

**Author:** ![pure-quetzal](https://avatars.discourse-cdn.com/v4/letter/p/ac91a4/32.png) [@pure-quetzal](https://community.cloudquery.io/u/pure-quetzal)\
**Post date:** [February 23, 2024, 7:27pm UTC](https://community.cloudquery.io/t/cloudquery-s3-destination-json-fields-landing-as-base64-encoded-is-it-configurable/386/4 "2024-02-23T19:27:47Z")

</div>

Hi folks,

Thank you for the responses, and I apologize for my own delay 🙂 Here is the configuration:

```yaml
kind: source
spec:
  name: "aws"
  path: "cloudquery/aws"
  registry: "cloudquery"
  version: "v23.6.1"
  ## See all tables: https://www.cloudquery.io/docs/plugins/sources/aws/tables
  tables: ["aws_s3_buckets", "aws_ec2_instances", "aws_ec2_images"]
  skip_tables:
    - "aws_s3_bucket_objects"
    - "aws_s3_bucket_object_grants"
  destinations: ["s3"]
  spec:
    aws_debug: true
    accounts:
      - id: <ID>
    regions:
      - "us-east-1"
      - "us-west-2"
---
kind: destination
spec:
  name: "s3"
  path: "cloudquery/s3"
  registry: "cloudquery"
  version: "v4.8.9"
  write_mode: "append"
  spec:
    bucket: <BUCKET>
    region: "us-east-1" # Example: us-east-1
    path: "raw/{{TABLE}}/{{UUID}}.parquet"
    format: "parquet" # options: parquet, json, csv
    format_spec:
      # CSV-specific parameters:
      # delimiter: ","
      # skip_header: false

    # Optional parameters
    athena: true # <- set this to true for Athena compatibility

```

We are observing this behavior both in the CloudQuery-provided AWS plugin, as well as a custom plugin we have implemented in Python.

Hi @herman, do you have any advice here? TYIA

---

<div class="post-metadata">

**Author:** ![yevgenyp](https://avatars.discourse-cdn.com/v4/letter/y/3be4f8/32.png) [@yevgenyp](https://community.cloudquery.io/u/yevgenyp)\
**Post date:** [February 27, 2024, 5:16pm UTC](https://community.cloudquery.io/t/cloudquery-s3-destination-json-fields-landing-as-base64-encoded-is-it-configurable/386/5 "2024-02-27T17:16:57Z")

</div>

Looping in @KernelKnight as Herman is out this week.

---

<div class="post-metadata">

**Author:** ![KernelKnight](https://avatars.discourse-cdn.com/v4/letter/k/9de053/32.png) [@KernelKnight](https://community.cloudquery.io/u/KernelKnight)\
**Post date:** [February 27, 2024, 6:06pm UTC](https://community.cloudquery.io/t/cloudquery-s3-destination-json-fields-landing-as-base64-encoded-is-it-configurable/386/6 "2024-02-27T18:06:23Z")

</div>

Hi all!

I’ll take a look and post here once I have some ideas.

OK, I do see a way to improve things: we use `Append` call on the bytes, which shouldn’t be, as the `Append` call for `JSONBuilder` marshals the data itself.

I’ll conjure a quick fix, but I’d like to give it for you to test (along with the change PR to see the changes, if you wish), so that we’re on the same page here.

---

<div class="post-metadata">

**Author:** ![pure-quetzal](https://avatars.discourse-cdn.com/v4/letter/p/ac91a4/32.png) [@pure-quetzal](https://community.cloudquery.io/u/pure-quetzal)\
**Post date:** [February 27, 2024, 6:41pm UTC](https://community.cloudquery.io/t/cloudquery-s3-destination-json-fields-landing-as-base64-encoded-is-it-configurable/386/7 "2024-02-27T18:41:54Z")

</div>

Yes! That would be fantastic! Thank you, Alex!

---

<div class="post-metadata">

**Author:** ![KernelKnight](https://avatars.discourse-cdn.com/v4/letter/k/9de053/32.png) [@KernelKnight](https://community.cloudquery.io/u/KernelKnight)\
**Post date:** [February 27, 2024, 6:43pm UTC](https://community.cloudquery.io/t/cloudquery-s3-destination-json-fields-landing-as-base64-encoded-is-it-configurable/386/8 "2024-02-27T18:43:30Z")

</div>

PR: [cloudquery/cloudquery/pull/16942](https://github.com/cloudquery/cloudquery/pull/16942)  
Version to try: `v4.10.2-rc`  
Tag for reference: [plugins-destination-s3-v4.10.2-rc](https://github.com/cloudquery/cloudquery/releases/tag/plugins-destination-s3-v4.10.2-rc)  
@pure-quetzal please let us know when you’ve run a sync & verified results 🥺

---

<div class="post-metadata">

**Author:** ![pure-quetzal](https://avatars.discourse-cdn.com/v4/letter/p/ac91a4/32.png) [@pure-quetzal](https://community.cloudquery.io/u/pure-quetzal)\
**Post date:** [February 27, 2024, 7:03pm UTC](https://community.cloudquery.io/t/cloudquery-s3-destination-json-fields-landing-as-base64-encoded-is-it-configurable/386/9 "2024-02-27T19:03:20Z")

</div>

Looks like I need to do `cloudquery login` now. Any guidance on how to achieve this in an AWS Airflow environment?

---

<div class="post-metadata">

**Author:** ![yevgenyp](https://avatars.discourse-cdn.com/v4/letter/y/3be4f8/32.png) [@yevgenyp](https://community.cloudquery.io/u/yevgenyp)\
**Post date:** [February 27, 2024, 7:04pm UTC](https://community.cloudquery.io/t/cloudquery-s3-destination-json-fields-landing-as-base64-encoded-is-it-configurable/386/10 "2024-02-27T19:04:36Z")

</div>

Can you create a token in the UI and set it as an environment variable in the Airflow environment?

[CloudQuery Documentation - Generate API Key](https://docs.cloudquery.io/docs/deployment/generate-api-key) I think should do it.

---

<div class="post-metadata">

**Author:** ![pure-quetzal](https://avatars.discourse-cdn.com/v4/letter/p/ac91a4/32.png) [@pure-quetzal](https://community.cloudquery.io/u/pure-quetzal)\
**Post date:** [February 27, 2024, 7:13pm UTC](https://community.cloudquery.io/t/cloudquery-s3-destination-json-fields-landing-as-base64-encoded-is-it-configurable/386/11 "2024-02-27T19:13:04Z")

</div>

This should work, thank you.  
I’m getting Firebase errors when attempting to create a CloudQuery Cloud account.

---

<div class="post-metadata">

**Author:** ![yevgenyp](https://avatars.discourse-cdn.com/v4/letter/y/3be4f8/32.png) [@yevgenyp](https://community.cloudquery.io/u/yevgenyp)\
**Post date:** [February 27, 2024, 7:20pm UTC](https://community.cloudquery.io/t/cloudquery-s3-destination-json-fields-landing-as-base64-encoded-is-it-configurable/386/12 "2024-02-27T19:20:58Z")

</div>

What error do you see?

---

<div class="post-metadata">

**Author:** ![pure-quetzal](https://avatars.discourse-cdn.com/v4/letter/p/ac91a4/32.png) [@pure-quetzal](https://community.cloudquery.io/u/pure-quetzal)\
**Post date:** [February 27, 2024, 7:22pm UTC](https://community.cloudquery.io/t/cloudquery-s3-destination-json-fields-landing-as-base64-encoded-is-it-configurable/386/13 "2024-02-27T19:22:35Z")

</div>

Firebase: Error (auth/internal-error).

---

<div class="post-metadata">

**Author:** ![yevgenyp](https://avatars.discourse-cdn.com/v4/letter/y/3be4f8/32.png) [@yevgenyp](https://community.cloudquery.io/u/yevgenyp)\
**Post date:** [February 27, 2024, 7:23pm UTC](https://community.cloudquery.io/t/cloudquery-s3-destination-json-fields-landing-as-base64-encoded-is-it-configurable/386/14 "2024-02-27T19:23:35Z")

</div>

Checking this out now. Are you trying to sign up via: [https://cloud.cloudquery.io/auth/register](https://cloud.cloudquery.io/auth/register)?

---

<div class="post-metadata">

**Author:** ![pure-quetzal](https://avatars.discourse-cdn.com/v4/letter/p/ac91a4/32.png) [@pure-quetzal](https://community.cloudquery.io/u/pure-quetzal)\
**Post date:** [February 27, 2024, 7:24pm UTC](https://community.cloudquery.io/t/cloudquery-s3-destination-json-fields-landing-as-base64-encoded-is-it-configurable/386/15 "2024-02-27T19:24:37Z")

</div>

The above error is with G Suite. When I try with email, I get the following error:

```auto
Firebase: Error (auth/firebase-app-check-token-is-invalid.).

```

---

<div class="post-metadata">

**Author:** ![yevgenyp](https://avatars.discourse-cdn.com/v4/letter/y/3be4f8/32.png) [@yevgenyp](https://community.cloudquery.io/u/yevgenyp)\
**Post date:** [February 27, 2024, 7:35pm UTC](https://community.cloudquery.io/t/cloudquery-s3-destination-json-fields-landing-as-base64-encoded-is-it-configurable/386/16 "2024-02-27T19:35:34Z")

</div>

We tried registering now on our end from different machines and it seems to work. Can you try from incognito? (Just to make sure there are no cache issues.) Also, which browser do you use? (We will try to replicate the issue.)

---

<div class="post-metadata">

**Author:** ![pure-quetzal](https://avatars.discourse-cdn.com/v4/letter/p/ac91a4/32.png) [@pure-quetzal](https://community.cloudquery.io/u/pure-quetzal)\
**Post date:** [February 27, 2024, 7:36pm UTC](https://community.cloudquery.io/t/cloudquery-s3-destination-json-fields-landing-as-base64-encoded-is-it-configurable/386/17 "2024-02-27T19:36:20Z")

</div>

Hm, okay. I’m using Chrome.

---

<div class="post-metadata">

**Author:** ![yevgenyp](https://avatars.discourse-cdn.com/v4/letter/y/3be4f8/32.png) [@yevgenyp](https://community.cloudquery.io/u/yevgenyp)\
**Post date:** [February 27, 2024, 7:36pm UTC](https://community.cloudquery.io/t/cloudquery-s3-destination-json-fields-landing-as-base64-encoded-is-it-configurable/386/18 "2024-02-27T19:36:46Z")

</div>

Maybe there are some extensions or something that are blocking something, but it’s pretty weird indeed.

---

<div class="post-metadata">

**Author:** ![pure-quetzal](https://avatars.discourse-cdn.com/v4/letter/p/ac91a4/32.png) [@pure-quetzal](https://community.cloudquery.io/u/pure-quetzal)\
**Post date:** [February 27, 2024, 7:37pm UTC](https://community.cloudquery.io/t/cloudquery-s3-destination-json-fields-landing-as-base64-encoded-is-it-configurable/386/19 "2024-02-27T19:37:33Z")

</div>

Same results with Incognito.  
I’m pretty sure it’s something on my end at this point.

---

<div class="post-metadata">

**Author:** ![yevgenyp](https://avatars.discourse-cdn.com/v4/letter/y/3be4f8/32.png) [@yevgenyp](https://community.cloudquery.io/u/yevgenyp)\
**Post date:** [February 27, 2024, 7:43pm UTC](https://community.cloudquery.io/t/cloudquery-s3-destination-json-fields-landing-as-base64-encoded-is-it-configurable/386/20 "2024-02-27T19:43:24Z")

</div>

I tried from both Chrome and Brave, but even if it’s on your end, then the message should be more descriptive, even though it’s a Firebase error, so we don’t control that part much. I’m still trying to investigate. Do you have Firefox or anything else? (Just trying to narrow down the cause of this.)

[Next page](https://community.cloudquery.io/t/cloudquery-s3-destination-json-fields-landing-as-base64-encoded-is-it-configurable/386.md?page=2)