Text transcode from source file member

[lucat73]

Type: Bug

Opening a RPGLE source file in a member with a CCSID = 280 the editor fail to translate some char with accents or particular symbols like § or à and replace them with <?>.

The iSeries connection user has profile CCSID 280 but i tried also with 1144 with no luck.
The iSeries OS is 7.3
The editor is set to UTF8.

Extension version: 1.6.13
VS Code version: Code 1.74.3 (97dec172d3256f8ca4bfb2143f3f76b503ca0534, 2023-01-09T16:59:02.252Z)
OS version: Windows_NT x64 10.0.19044
Modes:
Sandboxed: Yes

System Info

Item	Value
CPUs	11th Gen Intel(R) Core(TM) i5-1135G7 @ 2.40GHz (8 x 2419)
GPU Status	2d_canvas: enabled canvas_oop_rasterization: disabled_off direct_rendering_display_compositor: disabled_off_ok gpu_compositing: enabled multiple_raster_threads: enabled_on opengl: enabled_on rasterization: enabled raw_draw: disabled_off_ok skia_renderer: enabled_on video_decode: enabled video_encode: enabled vulkan: disabled_off webgl: enabled webgl2: enabled webgpu: disabled_off
Load (avg)	undefined
Memory (System)	7.85GB (1.61GB free)
Process Argv	--crash-reporter-id 47352308-b063-472e-8ccd-00c193597573
Screen Reader	no
VM	0%

A/B Experiments

vsliv368cf:30146710
vsreu685:30147344
python383:30185418
vspor879:30202332
vspor708:30202333
vspor363:30204092
vswsl492:30256859
vslsvsres303:30308271
pythonvspyl392:30443607
vserr242cf:30382550
pythontb:30283811
vsjup518:30340749
pythonptprofiler:30281270
vshan820:30294714
vstes263cf:30335440
vscorecescf:30445987
pythondataviewer:30285071
vscod805:30301674
binariesv615:30325510
bridge0708:30335490
bridge0723:30353136
cmake_vspar411:30581797
vsaa593cf:30376535
pythonvs932:30410667
cppdebug:30492333
vsclangdf:30486550
c4g48928:30535728
dsvsc012cf:30540253
azure-dev_surveyone:30548225
pyindex848cf:30577861
nodejswelcome1cf:30587006
282f8724:30602487
pyind779cf:30647092
f6dab269:30613381
pythonsymbol12cf:30652349
2i9eh265:30646982
vscsb:30628654

[worksofliam]

Hi @lucat73

I would suggest you ensure that your IBM i is configured properly.

Check out our docs to see how to do that: https://halcyon-tech.github.io/docs/#/pages/tips/ccsid

People have this all the time and it's an easy fix: #1032

[lucat73]

Hi @worksofliam
I checked up the configuration following the suggested guide and it's ok.
The translation fails only when I set the Enable Source Date in the extension settings otherwise it's correct.

Reading the temp files the extension stores in /tmp directory seems to me that the data exchange with the iSeries happens by SQL statement not by copy so could it be some trouble due to SQL conversion?

Moreover when writing àèìòù char in VSC and saving to iSeries SRC member they are correctly transcoded, instead when the same member is reopened or refreshed by VSC the transcoding fails.

Any suggestion is very appreciated.
Thank you.

[worksofliam]

Hi!

If you've followed the configuration, then you should have no issue.

What version of the OS are you on?

@chrjorgensen any ideas?

[lucat73]

I'm on 7.3

[lucat73]

Hi @worksofliam, thanks for the quick reply.

I followed the docs also setting up the env variables for PASE as:

but still not be able to transcode correctly.

[chrjorgensen]

I can't recreate the problem here - we need more info...

To reduce the debug output, first copy the source file and member into a new source file and delete all lines in the member except the first 20 lines (containing the special characters). Then create a new filter in VS Code having only that source file.

After this, open and clear the output channel for Code for IBM i:

Then click on the member and open it and show us all the output from this channel.
Also show us the output from DSPPFM / and F10 - to see the hex values of the special characters - like this:

                         Display Physical File Member                                                                    
File . . . . . . :   CCSID280            Library  . . . . :   zzzzzzzzzz
Member . . . . . :   MEMBER              Record . . . . . :   1                                                          
Control  . . . . .                       Column . . . . . :   1                                                          
Find . . . . . . .                                                                                                       
* . . .  + . . .  . 1 . .  . . + .  . . . 2  . . . .  + . . .  . 3 . .  . . + .     *...+....1....+....2....+....3....+. 
F0F0F0F1 F0F0F2F3 F0F2F0F9 7C40E388 89A24093 89958540 8881A240 81409781 99818799   *000100230209§ This line has a paragr*
F0F0F0F2 F0F0F2F3 F0F2F0F9 E38889A2 40938995 85408881 A240A388 85408481 9589A288   *000200230209This line has the danish*
F0F0F0F3 F0F0F2F3 F0F2F0F9 E38889A2 40938995 85408881 A2408388 81998183 A3859940   *000300230209This line has character *
F0F0F0F4 F0F0F2F3 F0F2F0F9 C0D0A16A 7940A681 A2408388 81958785 84404040 40404040   *000400230209àèìòù was changed       *
                          ****** END OF DATA ******                                                                      

Also provide us with the output from DSPUSRPRF page 5 - mine looks like this (while testing):

                         Display User Profile - Basic  
                                                       
User profile . . . . . . . . . . . . . . . :   zzzzzzzzzz
                                                       
Special environment  . . . . . . . . . . . :   *SYSVAL 
Attention program  . . . . . . . . . . . . :   QUSCMDLN
  Library  . . . . . . . . . . . . . . . . :     *LIBL 
Sort sequence  . . . . . . . . . . . . . . :   *SYSVAL 
  Library  . . . . . . . . . . . . . . . . :           
Language identifier  . . . . . . . . . . . :   ITA     
Country or region identifier . . . . . . . :   IT      
Coded character set identifier . . . . . . :   280     
Character identifier control . . . . . . . :   *SYSVAL 
Locale job attributes  . . . . . . . . . . :   *SYSVAL

[lucat73]

To keep things simple the only member of the SRCF has only this 3 rows:

Once opened in VSC the source is this:

and the output of the channel Code for IBM i is this:

`
/home/riwedp04: LC_ALL=EN_US.UTF-8 system "call QSYS/QZDFMDB2 PARM('-d' '-i' '-t')"
CREATE OR REPLACE ALIAS YLT.YLT_SRCPGM_TEST for "YLT"."SRCPGM"("TEST")
{
"code": 0,
"signal": null,
"stdout": "DB2>\n ?>\nDB20000I THE SQL COMMAND COMPLETED SUCCESSFULLY.",
"stderr": ""
}

/home/riwedp04: LC_ALL=EN_US.UTF-8 system "call QSYS/QZDFMDB2 PARM('-d' '-i' '-t')"
SELECT LENGTH(srcdta) as LENGTH FROM YLT.YLT_SRCPGM_TEST limit 1
{
"code": 0,
"signal": null,
"stdout": "DB2>\n ?>\n\nLENGTH \n-----------\n 100\n\n 1 RECORD(S) SELECTED.",
"stderr": ""
}

/home/riwedp04: LC_ALL=EN_US.UTF-8 system "call QSYS/QZDFMDB2 PARM('-d' '-i' '-t')"
select srcdat, rtrim(srcdta) as srcdta from YLT.YLT_SRCPGM_TEST
{
"code": 0,
"signal": null,
"stdout": "DB2>\n ?>\n\nSRCDAT SRCDTA \n-------- ----------------------------------------------------------------------------------------------------\n 230209 test1 \n 230208 � � � � � � \n 230209 test3 \n\n 3 RECORD(S) SELECTED.",
"stderr": ""
}
`

Output of DSPPFM:

Output on DSPUSRPRF page 5:

[chrjorgensen]

Thanks - and what is the CCSID on the source file DSPFD YLT/SRCPGM?

[lucat73]

It's 280.

[chrjorgensen]

Sorry, I meant DSPFFD YLT/SRCPGM.

And can you find the SSH job servicing your connection - probably a job called QP0ZSPWP in subsystem QUSRWRK (tip: find the job that started when you connected in VS Code) - and send the result of DSPJOB zzz OUTPUT(*ENVVAR)?

[lucat73]

DSPFFD YLT/SRCPGM => CCSID 280

DSPJOB zzz OPTION(*ENVVAR)

[lucat73]

I run the command directly in QSHELL and it seems ok:

I don't know if the lack of EN_UTF8 locale it's a problem, but the output of the SQL statement is correct.
Could it be a conversion problem of Visual Studio Code or the Code for i extension?

[worksofliam]

@lucat73 Glad to hear it!!

[vuxat]

Hi,

how about using a CAST to UTF-8 in the SQL Queries? (for source files and library / member descriptions)

e.g. /src/filesystems/qsys/complex/content.js - downloadMemberContentWithDates

    let rows = await content.runSQL(
      `select srcdat, rtrim(srcdta) as srcdta from ${aliasPath}`
    );

=>

 let rows = await content.runSQL(
      `select srcdat, rtrim(cast(srcdta char(${recordLengths[alias]}) CCSID 1208)) as srcdta from ${aliasPath}`
 );

I also have the same problem with "special characters" in member descriptions, source comments and literals/constants, but only when I switch on SQL support and/or Source with Dates. (QCCSID 273, German => e.g. ÄÖÜäöüß)

I do not know if uploadMemberContentWithDates is able to handle it correctly without changes. IBM i should know the data is UTF-8 and the target file CCSID is not UTF-8.

Best regards,
Stefan

[worksofliam]

@vuxat good idea. The downside about the cast to UTF8 is that it won't work when the system CCSID is 65535, which is a problem we are facing more often than not.

Does this cast still work when the system settings are not configured correctly?

[chrjorgensen]

@vuxat

I also have the same problem with "special characters" in member descriptions, source comments and literals/constants, but only when I switch on SQL support and/or Source with Dates. (QCCSID 273, German => e.g. ÄÖÜäöüß)

Some of the SQL columns for object and member descriptions are encoded in 65535 in the system and cannot be cast to UTF-8... however, we have built the conversion in this extension , so if your user profile / system and SSH daemon has the correct settings, you should get the correct characters. Please follow the guidelines in https://halcyon-tech.github.io/docs/#/pages/tips/ccsid.

[worksofliam]

@chrjorgensen i completely forgot about the fact that people also have source files with CCSID 65535... 😅😅

[vuxat]

@chrjorgensen @worksofliam unfortunately in my case I cannot set the system CCSIDs (SYSVAL, ENVVAR for SSH, FTP, ...) villy nilly without breaking old grimy stuff because there are many "grown" programs of 30 years which had/have to work with a SYSVAL of 65535 which is lol and sad.

At the moment I am happy that I set my user profile to a different CCSID and
found some hacky stuff to convert single byte CCSID on the fly:
(replace CCSIDs as necessary)

dcl-ds dataconv;
    datain char(2000) cssid(37) pos(1); //or 500/819 for data from ftp files
    dataout char(2000) ccsid(273) pos(1);
dcl-ds;
datain = OLDCCSIDDATA;
NEWCCSIDDATA = dataout;

The funny thing is without SQL enabled the *FILE variable knows, that the source file is 273 and converts it correctly for displaying, changing and saving.

How about still use the cast but with 65535 as default, with an overrule variable in the settings?
SQL should not react to 65535 at all so for most of the users it should be safe and for us poor europeans it could help a little.

I tried it with the IBMiContent Aliasing/select statement - viewed in Hex, but I have no source file in 65535 ...

select srcdat, rtrim(CAST(rtrim(srcdta) AS char(100) CCSID 65535)) as srcdta, HEX(rtrim(CAST(rtrim(srcdta) AS char(100) CCSID 65535))) as srcdtah from ...
select srcdat, rtrim(CAST(rtrim(srcdta) AS char(100) CCSID 1208)) as srcdta, HEX(rtrim(CAST(rtrim(srcdta) AS char(100) CCSID 1208))) as srcdtah from ...

Documentation like 'try first with CCSID 37 as cast overrule because it is the default codepage for data saved as CCSID 65535'
Default 'cast overrule' variable = empty

[worksofliam]

This kind of change would need a lot of testing.

It took a while to get the CCSID handling right and documented correctly. I will have a play with this but I can't guarantee that anything will change.

Cc @chrjorgensen

[chrjorgensen]

@worksofliam
I agree, we should be very careful if we change anything in the current setup.
AFAIK, all conversion problems reported by users have been caused by some wrong user or system CCSID and/or PASE/SSHD language environment variable, and the users were able to get the expected result by setting the correct values.

[vuxat]

@worksofliam @chrjorgensen

Hi,

I think I found another solution.

I went through the data transfer commands and found, that my problems arise during the sftp file transfer.
CPYTOSTMF FROMMBR('${path}') TOSTMF('${tempRmt}') STMFOPT(*REPLACE) STMFCCSID(1208) DBFCCSID(${this.config.sourceFileCCSID})
Creates a correct UTF-8 file
Transfering the file with client.getfile the data is again translated to my CCSID 273 (user settings, utf-8 not standard on our server)
I then sent the ftp command
quote type c 1208
tried again and also the transfered file was in utf-8

My proposed solution would be:
/src/api/IBMi.ts
after a successful async connect
to add the following command:

const commandResult = await this.sendCommand({
    command: `quote type c 1208`
});

Would that be possible without problems for the other users?

Thank you.

Best regards,
Stefan

[vuxat]

@worksofliam @chrjorgensen maybe a correction - you could use the command without the 'quote' - it depends on the used ftp library.
FTP answer:
200 Representation type is CCSID 1208.

[worksofliam]

As a side note, we don't use FTP, we use SFTP, which doesn't include anything to translate files.

[vuxat]

hi,
@worksofliam @chrjorgensen
ok, sftp would transfer differently. So i searched deeper.
Turns out, runSQL() is my personal culprit.

My user is set to CCSID 273. When this code is being executed
LC_ALL=EN_US.UTF-8 system "call QSYS/QZDFMDB2 PARM('-d' '-i' '-t')"
it does not create a UTF-8 data for me, just saves it again as 273.
Even tried casting in the call itself - cast and hex(cast) would show, that the normal call would be converted while the hex cast shows the UTF-8 chars C2xx ...) (directy and redirected to a file)
So I played around until I gave up, downloaded your source, installed the whole nodejs/npm/webpack stuff and changed the following parts in IBMIContents.ts and it works for me. (running with F5)
It reads and writes UTF-8 source files without problems. Compiling etc..
Source Dates and Source Dates in Gutter do not make any trouble.

A bit brute force ....

command:LC_ALL=EN_US.UTF-8 system "call QSYS/QZDFMDB2 PARM('-d' '-i' '-t')",
to
command:LC_ALL=EN_US.UTF-8 system "call QSYS/QZDFMDB2 PARM('-d' '-i' '-t')" | iconv -f ISO-8859-1 -t UTF-8,
(I know, nicer solution would have been to use the existing convertToUTF8() and the correct ccsid name instead of iso-8859-1)

And also this part (because 819 and 1252 are not UTF-8? I do not know the reason for their inclusion but for me 819/1252 are ISO-8859-1/15 which breaks data read and saved again as UTF-8 files)

const UTF8_CCSIDS = [819, 1208, 1252];
to
const UTF8_CCSIDS = [1208];

[chrjorgensen]

@vuxat We have users with all kind of CCSID's, including 273, who don't have the problems you experience... so I would suggest you follow the guidelines in https://halcyon-tech.github.io/docs/#/pages/tips/ccsid, especially the notes about SSH daemon and setting the environment variables. With the correct CCSID and SSH daemon settings the server will convert correctly from EBCDIC to UTF-8.

We have seen several issues filed by users about conversion gone wrong - and it always falls back to the server settings and mostly the SSH daemon language settings. Once the server is configured correctly, the conversion works as expected and we close the issues. We have done extensive testing in different setups with many CCSID and SSH daemon settings, and the code is the result of these tests.

We understand it can sometimes be difficult to change settings on the server, and we provide some settings in our code to circumvent this, like disabling SQL and force a CCSID value. But your server does not have the correct settings and we can't make special code just to accommodate your server settings. You seem to have found a solution, which works for you, and you can keep running your fork of this extension and merge any updates in the future, if you so wish.

Btw, if you get CCSID 273 output from the call to QSYS/QZDFMDB2 you will see garbage characters in the output window for Code for IBM i, since it expects UTF-8 character encoding. The SQL utility is used in many places besides reading and updating source members, so this should be very clear to see in the output window. If you see no garbage characters in the output window, the output from QSYS/QZDFMDB2 is converted correctly.

We hope you will find a solution that makes Code for IBM i work for you.

[vuxat]

@chrjorgensen yes, unfortunately I get garbage characters with the unchanged version. Also in the sources with comments and hardcoded variables (..oh no...).

Thank you anyway - I hope for a better setup of our system in the next year or two lol