On Thu, 9 Jan 2025 03:57:14 -0000 (UTC), Rich wrote:

>Display depends upon whether your font being used had a glyph for the
>codepoint - no glyph in the font, no display in the text widget

>That also depends upon what your system encoding was set to, and
>
>That is probably when support for the extended Unicode characters
>(planes beyond the BMP) started to be added.
>
**************************

Nothing to do with fonts or encoding. The problem vanished as soon as
I used 8.6.13, later 8.6.15. It was extended Unicode characters.

You can see my discussion here, at the end of the page:

https://wiki.tcl-lang.org/page/Unicode

--
Luc
>>

Folks,

thanks for all your suggestions and discussions.

I think this discussion has helped to bring the topic
into focus. May be that other Tcl users will be affected
by this incompatibility when migrating to Tcl9.
We have now examined the problem from many different
angles and have come up with more or less elaborate solutions.

My conclusion is: If migrating to Tcl9 all your source files
have to be encoded in utf-8. Otherwise you will have much more
maintaining efforts (adding iso8859, redefine commands, roll your own tcl, ...).

This can be mitigated if we introduce one of the changes I have proposed:
1. a switch (environment variable?) that restores the Tcl8 behavior and/or
2. a magic comment within the source file that can be used to determine the encoding.

I try to file a ticket about this.

Nevertheless, this point should be noted under "Important Incompatibilities in Tcl 9.0"
on the Tcl9 page:
https://www.tcl.tk/software/tcltk/9.0.html

Thanks again for your time!
Happy Tcl'ing
Uwe

Am 09.01.2025 um 05:15 schrieb Luc:
> On Thu, 9 Jan 2025 03:57:14 -0000 (UTC), Rich wrote:
>
>> Display depends upon whether your font being used had a glyph for the
>> codepoint - no glyph in the font, no display in the text widget
>
>> That also depends upon what your system encoding was set to, and
>>
>> That is probably when support for the extended Unicode characters
>> (planes beyond the BMP) started to be added.
>>
> **************************
>
> Nothing to do with fonts or encoding. The problem vanished as soon as
> I used 8.6.13, later 8.6.15. It was extended Unicode characters.
>
> You can see my discussion here, at the end of the page:
>
> https://wiki.tcl-lang.org/page/Unicode
>

Rich,

at first, thank you very much for explaining my situation very well.
I couldn't have argued better ;-)

Let me add a note on why characters outside the iso7-bit range
cannot always be replaced by the \uXXXX notation:
Comments.
If you like to write comments in your native language it
should be no very readable to code e.g. german umlauts as \uXXXX.
Especially if you extract the program documentation out of from
the source code in a kind of “literate programming” (which I often
do), the use of \u notation is very cumbersome.

Best wishes
Uwe

Am 08.01.2025 um 23:53 schrieb Rich:
> Luc <luc@sep.invalid> wrote:
>> On Wed, 8 Jan 2025 19:32:24 -0000 (UTC), Rich wrote:
>>
>>>> Instead of main.tcl sourcing set_encoding.tcl, starter.tcl runs some
>>>> 'encoding' command then sources main.tcl. Basically, a wrapper.
>>>
>>> Yes, that works. But then Uwe has to go and "wrapperize" all the
>>> various scripts, on all the various client systems. So he's back in
>>> the same boat of "major modifications need be made now" as changing all
>>> the launching instances to launch with "-encoding iso-8859".
>>
>> True, but he has considered that kind of effort. His words:
>>
>>
>> "That means we have to add "-encoding iso8859-1"
>> to ALL source and ALL tclsh calls in ALL scripts.
>> So far, so good(or bad?)."
>>
>> "What initially seems quite doable, looks more and more scary
>> to me. First, if we ever may switch encoding to utf-8 we
>> have to alter all those lines again."
>>
>>
>> So in my mind, the "customer" accepts (though grudgingly) making
>> large scale changes, but is concerned with possible new changes
>> in the future. A wrapper can handle the future quite gracefully.
>
> Uwe's reality is likely that at some point a "mass migration" may very
> well have to be done. There's at least two possibilities:
>
> 1) Tcl9 remains as it is today, loading all scripts as UTF-8 unless
> told otherwise by a user provided option. Either all iso-8859 scripts
> have to be modifed to become:
> 1a) UTF-8 encoded;
> 1b) modified pass the -encoding parameter to [source]
> 1c) a wrapper deployed that 'adjusts' things such that the main
> script, and all sourced scripts use -encoding to source as iso-8859
>
> All appear to be substantial work based on Uwe's statements so far, and
> all have a risk of overlooking one or more that should have been
> modified.
>
> 2) Tcl9 patch X reverts to using "system encoding" (and the user's of
> these scripts are on systems where "system encoding" is presently
> returning iso-8859). So things work again, with no changes, for the
> moment. But then Windows version 1Y.Z changes things such that it now
> uses a system encoding of UTF-8. Suddenly, the same problem from 1
> returns unless the user's have the abilty to adjust their system
> encoding back (and if 'system encoding' is an "administrator
> controlled" setting these users, then this option is not available).
>
> So my two cents, for what it is worth, given that I suspect this change
> will eventually 'force itself' no matter what Tcl9 patch level X might
> do, would be to begin the process of migrating all of these scripts to
> UTF-8 encoding. It will be hard, but once done, it likely will be
> stable again for the future.
>
>>> I've resisted pointing this one out, but long term, yes, updating all
>>> the scripts to be utf-8 encoded is the right, long term, answer. But
>>> that belies all the current, short term effort, involved in doing so.
>>
>> Actually, when I mentioned my migration case, I was also thinking that
>> I could afford to do it because I was migrating to Linux and utf-8 was
>> not even the future anymore, it was pretty much the present. But maybe
>> running iconv wouldn't be acceptable because Uwe is (I assume) on
>> Windows.
>
> From his posts on this thread, we can assume that his scripts are being
> used on windows systems. That does not imply much about where Uwe
> develops those same scripts. I have lots of my own scripts that I use
> on $work's windows machine, but all of them are written on Linux.
>
>> Does a Windows user want to convert his files to utf-8?
>
> The average/median windows user does not even know what UTF-8 means nor
> why it is significant. They just expect that when the launch "icon X"
> that expected program X appears, and that the text inside is as
> expected. So it is much more likely the work/effort of "convert to
> utf-8" will fall on Uwe, as it is very likely the windows users know
> nothing of any of this (or if they 'know' anything, it is something
> simple for them, such as: "set this selection box in this windows
> config pane to say Y" and that ends their knowledge).
>
>> Won't that cause problems if the system is iso-8859-1?
>
> Only if windows tries to interpret the UTF-8 data as iso-8859
> characters. But as far as the Tcl scripts go, once the scripts are
> UTF-8, and [source] is using UTF-8 to read them, the fact that windows
> system might be iso-8859 is irrelivant.
>
>> Windows still uses iso-8859-1, right?
>
> Honestly I have no idea. The *only* windows machine I use is $work's
> windows machine, and the 'administrator' controls most of it so I can
> only adjust things in a very narrow band (very irritating at times, but
> their machine, their rules).
>
>> So yes, I guess Tcl9 causes trouble to 8859-1 users.
>
> Only if they directly entered any codepoints that were beyond plain
> ASCII. Code points 0 through 127 are identical between 8859 and UTF-8.
> If the files used plain ASCII, and the \uXXXX escapes, there would be
> no trouble at all. Of course if one is using a lot of non-English
> characters for non-English languages, seeing the actual characters in
> the scripts vs. walls of \u00b0 \u00a0 \u2324 everywhere makes for an
> easier development effort.
>
>> Yes, sounds like it needs some fixing.
>
> Agreed. Uwe may be able to put off the fixig for some more time, but
> this change is going to arrive one day. He will likely have to make it
> at some point.
>
>> I have my own special case, I use Debian 9 which only ships 8.6.6 so
>> I had to build 8.6.15 from source because I really need Unicode.
>
> 8.6.6 handled Unicode fine. In fact, 8.5 handled Unicode (so long as
> one stuck to the BMP) just fine.
>
>> But for some time I used Freewrap as a single-file batteries included
>> Tcl/Tk interpreter. So maybe Uwe should just use a different interpreter,
>> likely just a slightly older version of Tcl/Tk and embrace Tcl9 later.
>
> That is another option, a custom build that defaults to iso-8859.
>
>> I wonder if one can hack the encoding issue on the Tcl9 source and
>> rebuild it.
>
> The answer is likely a "yes". But I've not looked at the code to know
> that for sure. But this just feels like a "one line change" followed
> by a recompile. But now one has to also deliver that custom runtime
> as well as the scripts that go with it.
>

Am 09.01.2025 um 10:12 schrieb Uwe Schmitz:
> Nevertheless, this point should be noted under "Important
> Incompatibilities in Tcl 9.0"
> on the Tcl9 page:
> https://www.tcl.tk/software/tcltk/9.0.html

Hi Uwe,
thanks for all your contributions.

Here is the wiki page for TCL script migration:

https://core.tcl-lang.org/tcl/wiki?name=Migrating+scripts+to+Tcl+9&p

Please look to section "Default encoding for scripts is UTF-8".

The also mentioned migration tools by Ashok also check the codepage
issue. You may consider to use those tools also to detect other
incompatible changes.
https://github.com/apnadkarni/tcl9-migrate

I am happy to include any missing information to this page.

Thank you and take care,
Harald

Uwe Schmitz <schmitzu@mail.de> wrote:
> Rich,
>
> at first, thank you very much for explaining my situation very well.
> I couldn't have argued better ;-)
>
> Let me add a note on why characters outside the iso7-bit range
> cannot always be replaced by the \uXXXX notation:
> Comments.

> If you like to write comments in your native language it
> should be no very readable to code e.g. german umlauts as \uXXXX.
> Especially if you extract the program documentation out of from
> the source code in a kind of “literate programming” (which I often
> do), the use of \u notation is very cumbersome.

This was my suspision. In my case, the non-ascii characters are not
part of the language (English in my case) script, they are extras (such
as arrows/lines or the degree symbol, etc.) and so the script is 99.9%
readable, with a few \uXXXX sometimes occurring.

But writing a string out where every third character is \uXXXX makes
for a very human unreadable string (be it a comment, or a string for
the code to use).

If you develop on Linux (or have a Linux machine available) you may
wish to begin experimenting with using iconv to convert some scripts to
UTF-8 encoding. If things work properly, it might be best to start
that conversion (even if you do it slowly over time) sooner rather than
later. It will be work, but it is work that you are likely going to
have to perform at some point anyway.

Luc <luc@sep.invalid> wrote:
> On Thu, 9 Jan 2025 03:57:14 -0000 (UTC), Rich wrote:
>
>>Display depends upon whether your font being used had a glyph for the
>>codepoint - no glyph in the font, no display in the text widget
>
>>That also depends upon what your system encoding was set to, and
>>
>>That is probably when support for the extended Unicode characters
>>(planes beyond the BMP) started to be added.
>>
> **************************
>
> Nothing to do with fonts or encoding. The problem vanished as soon as
> I used 8.6.13, later 8.6.15. It was extended Unicode characters.
>
> You can see my discussion here, at the end of the page:
>
> https://wiki.tcl-lang.org/page/Unicode

The answer confirmed what I said earlier:

"8.6 does not support characters above BMP without a little bit of
hackery"

And you were trying to make use of a 1F4C4 character, which is outside
the BMP.

On Thu, 9 Jan 2025 15:37:22 -0000 (UTC), Rich wrote:

>If you develop on Linux (or have a Linux machine available) you may
>wish to begin experimenting with using iconv to convert some scripts to
>UTF-8 encoding.

A quick search shows there is iconv for Windows.

> If things work properly, it might be best to start
>that conversion (even if you do it slowly over time) sooner rather than
>later. It will be work, but it is work that you are likely going to
>have to perform at some point anyway.

Yes, but now I think that Tcl9 is wrong. Blanket imposition of any
enconding is unfair.

--
Luc
>>

Luc <luc@sep.invalid> wrote:
> On Thu, 9 Jan 2025 15:37:22 -0000 (UTC), Rich wrote:
>
>>If things work properly, it might be best to start that conversion
>>(even if you do it slowly over time) sooner rather than later. It
>>will be work, but it is work that you are likely going to have to
>>perform at some point anyway.
>
> Yes, but now I think that Tcl9 is wrong. Blanket imposition of any
> enconding is unfair.

Tcl has to do one of two things:

1) Pick a default it will use.

2) Use the "system encoding" (which is still 'imposing', just
'imposing' whatever the OS itself imposes).

But there always /is/ an encoding being used, because there is no way
to process a textual file otherwise. Some "encoding" has to be chosen
to use to decode the characters in text files.

On Fri, 10 Jan 2025 00:12:48 -0000 (UTC), Rich wrote:

>2) Use the "system encoding" (which is still 'imposing', just
>'imposing' whatever the OS itself imposes).

Is the OS really imposing though? I honestly don't know about Windows,
but Linux lets me choose the system-wide encoding. And whatever I
chose, I must've chosen it for some reason. It's not Tcl's place
to challenge my decision.

And if the poor sorry Windows user really can't choose his encoding,
then why should Tcl make the user's life even more difficult?

The 8.6 way is wiser.

--
Luc
>>

On Fri, 10 Jan 2025 3:53:53 +0000, Luc wrote:

> On Fri, 10 Jan 2025 00:12:48 -0000 (UTC), Rich wrote:
>
>>2) Use the "system encoding" (which is still 'imposing', just
>>'imposing' whatever the OS itself imposes).
>
> Is the OS really imposing though? I honestly don't know about Windows,
> but Linux lets me choose the system-wide encoding. And whatever I
> chose, I must've chosen it for some reason. It's not Tcl's place
> to challenge my decision.
>
> And if the poor sorry Windows user really can't choose his encoding,
> then why should Tcl make the user's life even more difficult?
>
> The 8.6 way is wiser.

Not sure... your choice of encoding may not be the one of your
application users.
In this case, your code may fail to load in the user's encoding choice.

Eric

--

Rich,

Yes, what you write in your last paragraph is the conclusion
from the entire discussion here. But I wouldn't have thought
of that at the start of the Tcl9 migration process.
But this is definitely the path to follow.

Best wishes
Uwe

Am 09.01.2025 um 16:37 schrieb Rich:
> Uwe Schmitz <schmitzu@mail.de> wrote:
>> Rich,
>>
>> at first, thank you very much for explaining my situation very well.
>> I couldn't have argued better ;-)
>>
>> Let me add a note on why characters outside the iso7-bit range
>> cannot always be replaced by the \uXXXX notation:
>> Comments.
>
>> If you like to write comments in your native language it
>> should be no very readable to code e.g. german umlauts as \uXXXX.
>> Especially if you extract the program documentation out of from
>> the source code in a kind of “literate programming” (which I often
>> do), the use of \u notation is very cumbersome.
>
> This was my suspision. In my case, the non-ascii characters are not
> part of the language (English in my case) script, they are extras (such
> as arrows/lines or the degree symbol, etc.) and so the script is 99.9%
> readable, with a few \uXXXX sometimes occurring.
>
> But writing a string out where every third character is \uXXXX makes
> for a very human unreadable string (be it a comment, or a string for
> the code to use).
>
> If you develop on Linux (or have a Linux machine available) you may
> wish to begin experimenting with using iconv to convert some scripts to
> UTF-8 encoding. If things work properly, it might be best to start
> that conversion (even if you do it slowly over time) sooner rather than
> later. It will be work, but it is work that you are likely going to
> have to perform at some point anyway.

Harald,

thanks for the wiki page. It has definitely a lot of
information.

May be, you (or I can do that too) may add to the "Default encoding..."
paragraph, that in the long term it's best to encode ALL tcl source
files in utf-8 to get out of this "-encoding ..." hell.

Another thing that hurts me and is off-topic here (sorry):
The changed variable name resolution also affects itcl::class
defintions. The following leads to an error:

::itcl::class A {
public common tclVersion $tcl_version
}

Because the ::itcl::class commands open a namespace, the resolution
of the global variable tcl_version doesn't succeed. You
have to use the complete path $::tcl_version.

Best wishes
Uwe

Am 09.01.2025 um 10:40 schrieb Harald Oehlmann:
> Am 09.01.2025 um 10:12 schrieb Uwe Schmitz:
>> Nevertheless, this point should be noted under "Important Incompatibilities in Tcl 9.0"
>> on the Tcl9 page:
>> https://www.tcl.tk/software/tcltk/9.0.html
>
> Hi Uwe,
> thanks for all your contributions.
>
> Here is the wiki page for TCL script migration:
>
> https://core.tcl-lang.org/tcl/wiki?name=Migrating+scripts+to+Tcl+9&p
>
> Please look to section "Default encoding for scripts is UTF-8".
>
> The also mentioned migration tools by Ashok also check the codepage issue. You may consider to use those tools also to detect other incompatible changes.
> https://github.com/apnadkarni/tcl9-migrate
>
> I am happy to include any missing information to this page.
>
> Thank you and take care,
> Harald

Am 10.01.2025 um 13:09 schrieb Uwe Schmitz:
> Another thing that hurts me and is off-topic here (sorry):
> The changed variable name resolution also affects itcl::class
> defintions. The following leads to an error:
>
> ::itcl::class A {
>    public common tclVersion $tcl_version
> }
>
> Because the ::itcl::class commands open a namespace, the resolution
> of the global variable tcl_version doesn't succeed. You
> have to use the complete path $::tcl_version.

Yes, that is intentional and important. And Ashoks migration tolls also
catch this.

The issue in TCL 8.6 is:

namespace eval test { set test A }

Works as follows:
- if test exists in the global namespace, the global is set.
- if test does not exist in the global namespace, a namespace variable
"test" is created.

This feature had the consequence, that namespace variables sometimes
overwrote global variables, but were intended as namespace variables.

In TCL 9, this will always address a namespace variable.

Take care,
Harald

On Fri, 10 Jan 2025 06:38:02 +0000, eric wrote:

>> The 8.6 way is wiser.

>Not sure... your choice of encoding may not be the one of your
>application users.
>In this case, your code may fail to load in the user's encoding choice.
**************************

The user's choice is and always will be a point of uncertainty.

Tcl9 introduces an additional uncertainty with the developer.

--
Luc
>>

Luc <luc@sep.invalid> wrote:
> On Fri, 10 Jan 2025 06:38:02 +0000, eric wrote:
>
>>> The 8.6 way is wiser.
>
>>Not sure... your choice of encoding may not be the one of your
>>application users.
>>In this case, your code may fail to load in the user's encoding choice.
> **************************
>
> The user's choice is and always will be a point of uncertainty.
>
> Tcl9 introduces an additional uncertainty with the developer.

Actually, 9 /reduces/ uncertianty.

Current method:

Developer: sets his system encoding to ISO-8859. Writes Tcl script,
and includes 8859 code points directly into the script. everything
works for developer, on his system.

User #1: sets his system encoding to CP437 (the original DOS character
encoding -- I needed to pick 'something' other than 8859). Downloads
"Developers" script from github, and launches it.

If Tcl interprets the script using the sysstem encoding, it will
interpret the ISO-8859 script as if it were DOS CP437 bytes. So many
extended language letters will instead become lots of line draw
characters (just one example). The user is disappointed, as the script
does not work "out of the box" for him.

New Tcl9 method:

All Tcl scripts are UTF-8, no exceptions (not really true, but close
enough).

Developer: Must create the script with UTF-8 encoding. Note, the
'developer' could continue to use 8859 for 'writing' things, they
simply must use iconv (or similar) to convert to UTF-8 before feeding
the script to the Tcl interpreter.

User #1: Continues setting his/her system encoding to CP437. But now,
they download developer's script from github, and when they launch it
with Tcl9, it is always interpreted as UTF-8. It "just works", and
User #1 sees the proper accented characters the developer put into the
script for prompts or other strings. User #1 did not have to do
anything, and the script worked "out of the box" for him/her.

With the Tcl9 method of "all scripts must be UTF-8" there is less
uncertianty, because the script will be interpreted using the same
encoding everywhere, no matter what odd local system setting any given
user may have chosen.