Unable to read Arabic content from queries

[mahmoudelatma]

Hi everyone,

I have a Redash installation via the following Docker-compose.yml file on windows server machine, from within the application Homepage after registration, I've created a Datasource to connect to a MS Sql Server DB which have a collation of Arabic_100_CI_AI ,
I used UTF-8 as character set and TDS version 7.0 as defaults, however the queries result set doesn't show Arabic letters it shows something like this "ÇáãÔÑæÚ", looking forward to your support..

The Docker-compose.yml

version: '2'

x-redash-service: &redash-service
  image: redash/redash:latest
  depends_on:
    - postgres
    - redis
  env_file: redash.env
  restart: always

services:
  server:
    <<: *redash-service
    command: server
    ports:
      - "5000:5000"
    environment:
      REDASH_WEB_WORKERS: 4
      LC_ALL: C.UTF-8

  scheduler:
    <<: *redash-service
    command: scheduler
    environment:
      QUEUES: "celery"
      WORKERS_COUNT: 1

  scheduled_worker:
    <<: *redash-service
    command: worker
    environment:
      QUEUES: "scheduled_queries,schemas"
      WORKERS_COUNT: 1

  adhoc_worker:
    <<: *redash-service
    command: worker
    environment:
      QUEUES: "queries"
      WORKERS_COUNT: 2

  redis:
    image: redis:latest
    restart: always

  postgres:
    image: postgres:latest
    env_file: redash.env
    restart: always
    networks:
      - default

  nginx:
    image: nginx:latest
    ports:
      - "80:80"
    depends_on:
      - server
    links:
      - server:redash
    restart: always

and the redash.env file is as follows:

REDASH_HOST=http://localhost/redash
PYTHONUNBUFFERED=0
REDASH_LOG_LEVEL=INFO
REDASH_REDIS_URL=redis://redis:6379/0
POSTGRES_PASSWORD=
POSTGRES_HOST_AUTH_METHOD=trust
REDASH_COOKIE_SECRET=####
REDASH_SECRET_KEY=####
REDASH_DATABASE_URL=postgresql://postgres@postgres/postgres

[mahmoudelatma]

I figured it out.. 😄 it looks like Redash were unable to read Arabic characters when the field type is VARCHAR, when I converted it to NVARCHAR, Arabic letters where visible successfully.

[justinclift]

That's VARCHAR in the MS SQL Server database yeah, not in the internal Redash database (PostgreSQL)?

[mahmoudelatma]

Yes, VARCHAR in MS SQL SERVER DB. Which i connected to it as a datasource, it looks like Arabic letters aren't decoded correctly for redash when reading from a field of this type, when I changed it to NVARCHAR, it was shown as expected.

[justinclift]

Cool. Thanks for adding the solution details here, so future people having the same kind of problem have a chance of finding it. 😄

[guidopetri]

It looks like it's an SQL thing:

Each varchar character uses 1 byte of storage and can represent 256 different characters. Each nvarchar character uses 2 bytes of storage and can represent 65,536 different characters. If you need to support text like arabic, chinese, or emojis, use nvarchar. If you only expect alpha numeric and want storage benefits, varchar is better.

source