Message-ID:

"Life, loathe it or ignore it, you can't like it." -- Marvin, "Hitchhiker's Guide to the Galaxy"

rocksolid / Rocksolid Nodes Help / Re: text encoding of umlauts in latest view broken

Subject: text encoding of umlauts in latest view broken
From: Marco Moock
Newsgroups: rocksolid.nodes.help
Organization: A noiseless patient Spider
Date: Sun, 25 Aug 2024 09:32 UTC

View all headers

Hello!
https://pi-dach.dorfdsl.de/rocksolid/overboard.php?thisgroup=de.alt.folklore.computer

The post itself looks fine when viewed directly, but in the "latest"
view, the encoding of special German characters is broken.

This affects
<AABmyvIvTIsAAAoF.A3.flnews@WStation7.micha.freeshell.org>
Message-ID: <slrnvckmpm.55rd.hjp-usenet4@trintignant.hjp.at>

--
kind regards
Marco

Subject: Re: text encoding of umlauts in latest view broken
From: Retro Guy
Newsgroups: rocksolid.nodes.help
Organization: Rocksolid Light
Date: Sun, 25 Aug 2024 12:05 UTC
References: 1

View all headers

On Sun, 25 Aug 2024 11:32:03 +0200, Marco Moock wrote:

> Hello!
> https://pi-dach.dorfdsl.de/rocksolid/overboard.php?thisgroup=de.alt.folklore.computer
>
> The post itself looks fine when viewed directly, but in the "latest"
> view, the encoding of special German characters is broken.
>
> This affects
> <AABmyvIvTIsAAAoF.A3.flnews@WStation7.micha.freeshell.org>
> Message-ID: <slrnvckmpm.55rd.hjp-usenet4@trintignant.hjp.at>

Thank you. I've just added the group to my test server and will get on it.

I appreciate feedback from users about character display issues as the
groups I read are all English (guess why)

The first user, years ago, that got me on the right path to displaying
multibyte characters properly was Russian and was very helpful (very
specific feedback) in getting things working.

Subject: Re: text encoding of umlauts in latest view broken
From: Marcel Logen
Newsgroups: rocksolid.nodes.help
Organization: Bureau Logen Bonn Germany
Date: Sun, 25 Aug 2024 13:12 UTC
References: 1 2

Path: eternal-september.org!news.eternal-september.org!feeder3.eternal-september.org!weretis.net!feeder8.news.weretis.net!reader5.news.weretis.net!news.tota-refugium.de!.POSTED!not-for-mail
From: 333200007110-0201@ybtra.de (Marcel Logen)
Newsgroups: rocksolid.nodes.help
Subject: Re: text encoding of umlauts in latest view broken
Date: Sun, 25 Aug 2024 15:12:24 +0200
Organization: Bureau Logen Bonn Germany
Message-ID: <AABmyy240AwAAPD3.A3.flnews@t20.ybtra.de>
References: <vaetmk$1q1gt$3@dont-email.me> <e2db830f8efe89dcea07f1bbb5ef6601$1@novabbs.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8; format=fixed
Content-Transfer-Encoding: 8bit
Injection-Date: Sun, 25 Aug 2024 15:12:45 +0200
Injection-Info: tota-refugium.de;
logging-data="1803639"; mail-complaints-to="abuse@news.tota-refugium.de"
User-Agent: flnews/1.3.0pre10 (for OpenBSD) ml-inews
Cancel-Lock: sha256:qjYJqkwzkiIP68ppSD4KGfWcw4D7FJzuoYQsfUm8bFM= sha1:fjJu0QFTiNGzD+bnLoi1tJOehVQ=
X-User-ID: eJwNyMEBwCAIA8CVjJCI41iQ/Uew9zyaoFwuytlszxrHOffcstP/gV+wuK7dFkpVSAxXoPPGAxWxEQg=

View all headers

Retro Guy in rocksolid.nodes.help:

>On Sun, 25 Aug 2024 11:32:03 +0200, Marco Moock wrote:

>> https://pi-dach.dorfdsl.de/rocksolid/overboard.php?thisgroup=de.alt.folklore.computer
>>
>> The post itself looks fine when viewed directly, but in the "latest"
>> view, the encoding of special German characters is broken.
>>
>> This affects
>> <AABmyvIvTIsAAAoF.A3.flnews@WStation7.micha.freeshell.org>
>> Message-ID: <slrnvckmpm.55rd.hjp-usenet4@trintignant.hjp.at>
>
>Thank you. I've just added the group to my test server and will get on it.
>
>I appreciate feedback from users about character display issues as the
>groups I read are all English (guess why)
>
>The first user, years ago, that got me on the right path to displaying
>multibyte characters properly was Russian and was very helpful (very
>specific feedback) in getting things working.

These articles use "charset=ISO-8859-1" for the body.

Marcel 1n1bn (1803639)
--
╭───╮ ╭────────╮ ╭──╮ ╭───╮ ╭──╮ ╭──
╭─╮ ╰─╮ ╰──╯ ╭────╯ ╭────╯ │ ╭─╮ │ ╰──╯ ╰─╮ ╭────╮ │
──╮ │ ╰────╯ ╰──╮ ╭─╯ ╰─╯ ╰──╯ ╭────────╯ ╰──╮ ╰───╯
╰───╯ ╰──╯ ╰──────────────╯ 0c3fbd

Subject: Re: text encoding of umlauts in latest view broken
From: Marco Moock
Newsgroups: rocksolid.nodes.help
Organization: A noiseless patient Spider
Date: Sun, 25 Aug 2024 13:29 UTC
References: 1 2 3

Path: eternal-september.org!news.eternal-september.org!.POSTED!not-for-mail
From: mm+usenet-es@dorfdsl.de (Marco Moock)
Newsgroups: rocksolid.nodes.help
Subject: Re: text encoding of umlauts in latest view broken
Date: Sun, 25 Aug 2024 15:29:54 +0200
Organization: A noiseless patient Spider
Lines: 12
Message-ID: <vafbki$1s7hg$1@dont-email.me>
References: <vaetmk$1q1gt$3@dont-email.me>
<e2db830f8efe89dcea07f1bbb5ef6601$1@novabbs.org>
<AABmyy240AwAAPD3.A3.flnews@t20.ybtra.de>
MIME-Version: 1.0
Content-Type: text/plain; charset=US-ASCII
Content-Transfer-Encoding: 7bit
Injection-Date: Sun, 25 Aug 2024 15:29:55 +0200 (CEST)
Injection-Info: dont-email.me; posting-host="8c0b78ae15d81c73da5a74c7a040c842";
logging-data="1973808"; mail-complaints-to="abuse@eternal-september.org"; posting-account="U2FsdGVkX19Ez3YrDTj6xmvzaSTYy8F6"
Cancel-Lock: sha1:fRQtEMoiFwS0dhSyidphJCo81Tg=

View all headers

On 25.08.2024 um 15:12 Uhr Marcel Logen wrote:

> These articles use "charset=ISO-8859-1" for the body.

Is that RFC-conform or is there an issue with the article itself?

--
kind regards
Marco

Send spam to 1724591544muell@cartoonies.org

Path: eternal-september.org!news.eternal-september.org!feeder3.eternal-september.org!weretis.net!feeder8.news.weretis.net!reader5.news.weretis.net!news.tota-refugium.de!.POSTED!not-for-mail
From: 333200007110-0201@ybtra.de (Marcel Logen)
Newsgroups: rocksolid.nodes.help
Subject: Re: text encoding of umlauts in latest view broken
Date: Sun, 25 Aug 2024 16:05:00 +0200
Organization: Bureau Logen Bonn Germany
Message-ID: <AABmyzoMiikAARGb.A3.flnews@t20.ybtra.de>
References: <vaetmk$1q1gt$3@dont-email.me> <e2db830f8efe89dcea07f1bbb5ef6601$1@novabbs.org> <AABmyy240AwAAPD3.A3.flnews@t20.ybtra.de> <vafbki$1s7hg$1@dont-email.me>
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8; format=fixed
Content-Transfer-Encoding: 8bit
Injection-Date: Sun, 25 Aug 2024 16:05:12 +0200
Injection-Info: tota-refugium.de;
logging-data="1771880"; mail-complaints-to="abuse@news.tota-refugium.de"
User-Agent: flnews/1.3.0pre10 (for OpenBSD) ml-inews
Cancel-Lock: sha256:wrJMrGeMjucC/vKx2pNAuvXsnScZtsYiDRkUlzhxJj0= sha1:qTmwt6ccIjE9RIWRGS6PeL79zHc=
X-User-ID: eJwNyNEVADEEBMCWCLtcOcLTfwm5+RwYlR1O0LHYGFyBoWP/YIpsxwQL4/DzqWmerKRzNW89BVsQIA==

View all headers

Marco Moock in rocksolid.nodes.help:

>On 25.08.2024 um 15:12 Uhr Marcel Logen wrote:

>> These articles use "charset=ISO-8859-1" for the body.
>
>Is that RFC-conform or is there an issue with the article itself?

The use of ISO-8859-1 is perfectly correct and RFC compliant,
but has become rather rare (in favor of UTF-8).

Marcel 1m2b8 (1771880)
--
╭───╮ ╭──╮ ╭─╮ ╭──────╮ ╭──╮ ╭──────╮
│ ╰───╯ │ │ ╰─╮ ╰────╮ ╰─────╯ ╰───╯ ╭───╯
╮ ╭─╮ ╭──╯ ╭─╯ ╭─╯ ╰──╮ ╭─╯ ╰───╮
╰────╯ ╰─────╯ ╰───╯ ╰──╯ b4a418 ╰──────

Path: eternal-september.org!news.eternal-september.org!feeder3.eternal-september.org!i2pn.org!i2pn2.org!.POSTED!not-for-mail
From: retroguy@novabbs.com (Retro Guy)
Newsgroups: rocksolid.nodes.help
Subject: Re: text encoding of umlauts in latest view broken
Date: Sun, 25 Aug 2024 08:13:13 -0700
Organization: Rocksolid Light
Message-ID: <cb45e76f63e84ef0c6588dc64c88a2fd$1@novabbs.org>
References: <vaetmk$1q1gt$3@dont-email.me> <e2db830f8efe89dcea07f1bbb5ef6601$1@novabbs.org> <AABmyy240AwAAPD3.A3.flnews@t20.ybtra.de> <vafbki$1s7hg$1@dont-email.me> <AABmyzoMiikAARGb.A3.flnews@t20.ybtra.de>
MIME-Version: 1.0
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: 7bit
Injection-Info: i2pn2.org;
logging-data="3855715"; mail-complaints-to="usenet@i2pn2.org";
posting-account="PGd4t4cXnWwgUWG9VtTiCsm47oOWbHLcTr4rYoM0Edo";
User-Agent: 40tude_Dialog/2.0.15.41
X-Spam-Checker-Version: SpamAssassin 4.0.0

View all headers

On Sun, 25 Aug 2024 16:05:00 +0200, Marcel Logen wrote:

> Marco Moock in rocksolid.nodes.help:
>
>>On 25.08.2024 um 15:12 Uhr Marcel Logen wrote:
>
>>> These articles use "charset=ISO-8859-1" for the body.
>>
>>Is that RFC-conform or is there an issue with the article itself?
>
> The use of ISO-8859-1 is perfectly correct and RFC compliant,
> but has become rather rare (in favor of UTF-8).

Yes, this charset is fine and should not be the problem.

I am working on this, but it will take some time. Part of the issue, but
not all of it, is decoding a portion of the body to create the "snippet". I
know it sounds simple to just decode the entire body and create the
snippet, but there is more to it than that.

It will take a bit of time to work out.

Path: eternal-september.org!news.eternal-september.org!feeder3.eternal-september.org!news.mixmin.net!news.neodome.net!rocksolid2!i2pn2.org!.POSTED!not-for-mail
From: retroguy@novabbs.com (Retro Guy)
Newsgroups: rocksolid.nodes.help
Subject: Re: text encoding of umlauts in latest view broken
Date: Sun, 25 Aug 2024 18:42:37 +0000
Organization: Rocksolid Light
Message-ID: <df91607f35e218a06711caea02ff4731@www.novabbs.org>
References: <vaetmk$1q1gt$3@dont-email.me> <e2db830f8efe89dcea07f1bbb5ef6601$1@novabbs.org> <AABmyy240AwAAPD3.A3.flnews@t20.ybtra.de> <vafbki$1s7hg$1@dont-email.me> <AABmyzoMiikAARGb.A3.flnews@t20.ybtra.de> <cb45e76f63e84ef0c6588dc64c88a2fd$1@novabbs.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Transfer-Encoding: 8bit
Injection-Info: i2pn2.org;
logging-data="3874933"; mail-complaints-to="usenet@i2pn2.org";
posting-account="gg+dDWHMzVrdxGO9Gmt8aqyeWDXqlxfqiuS0jX/WMXY";
User-Agent: Rocksolid Light
X-Spam-Checker-Version: SpamAssassin 4.0.0
X-Rslight-Site: $2y$10$/GMcGYi5ha5FEqaJRbZWGuk.CayX9QElz2SZYDWUh4/9q7fWbOBRa
X-Face: .&YR-G(w(DZ$$,}%k=]*5*!p'=(anr"IT`wZG'2VWdfl\r)l[42u7JH`n(JUQ*e5*A|XCDf
?&\X&uwkl38"CYX3O8m}C8E4p'%N$2#kSTVzx{Ly|DjLT\Vk7NE}NQ(VC$Yq]i:7|z[.9iv^g>*8_B
H0=hZt'[%)4kG|
X-Rslight-Posting-User: a93aefeeff923def71455caae2dbfb277a59e046

View all headers

On Sun, 25 Aug 2024 15:13:13 +0000, Retro Guy wrote:

> On Sun, 25 Aug 2024 16:05:00 +0200, Marcel Logen wrote:
>
>> Marco Moock in rocksolid.nodes.help:
>>
>>>On 25.08.2024 um 15:12 Uhr Marcel Logen wrote:
>>
>>>> These articles use "charset=ISO-8859-1" for the body.
>>>
>>>Is that RFC-conform or is there an issue with the article itself?
>>
>> The use of ISO-8859-1 is perfectly correct and RFC compliant,
>> but has become rather rare (in favor of UTF-8).
>
> Yes, this charset is fine and should not be the problem.
>
> I am working on this, but it will take some time. Part of the issue, but
> not all of it, is decoding a portion of the body to create the
> "snippet". I
> know it sounds simple to just decode the entire body and create the
> snippet, but there is more to it than that.
>
> It will take a bit of time to work out.

Does this look better? https://postimg.cc/Zv4gwcVP

or did I make it worse somehow? :)

--
Retro Guy

Path: eternal-september.org!news.eternal-september.org!.POSTED!not-for-mail
From: mm+usenet-es@dorfdsl.de (Marco Moock)
Newsgroups: rocksolid.nodes.help
Subject: Re: text encoding of umlauts in latest view broken
Date: Sun, 25 Aug 2024 21:49:46 +0200
Organization: A noiseless patient Spider
Lines: 12
Message-ID: <vag1sr$21ttr$1@dont-email.me>
References: <vaetmk$1q1gt$3@dont-email.me>
<e2db830f8efe89dcea07f1bbb5ef6601$1@novabbs.org>
<AABmyy240AwAAPD3.A3.flnews@t20.ybtra.de>
<vafbki$1s7hg$1@dont-email.me>
<AABmyzoMiikAARGb.A3.flnews@t20.ybtra.de>
<cb45e76f63e84ef0c6588dc64c88a2fd$1@novabbs.org>
<df91607f35e218a06711caea02ff4731@www.novabbs.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=US-ASCII
Content-Transfer-Encoding: 7bit
Injection-Date: Sun, 25 Aug 2024 21:49:47 +0200 (CEST)
Injection-Info: dont-email.me; posting-host="8c0b78ae15d81c73da5a74c7a040c842";
logging-data="2160571"; mail-complaints-to="abuse@eternal-september.org"; posting-account="U2FsdGVkX19NBYhFfof7v/6aea4/UOB9"
Cancel-Lock: sha1:/DudsyZDsKkGZdJ8IBna2TUt1xs=

View all headers

On 25.08.2024 um 18:42 Uhr Retro Guy wrote:

> Does this look better? https://postimg.cc/Zv4gwcVP

Looks fine, umlauts are now working.

--
kind regards
Marco

Send spam to 1724604157muell@cartoonies.org

Path: eternal-september.org!news.eternal-september.org!feeder3.eternal-september.org!news.mixmin.net!weretis.net!feeder9.news.weretis.net!i2pn.org!i2pn2.org!.POSTED!not-for-mail
From: retroguy@novabbs.com (Retro Guy)
Newsgroups: rocksolid.nodes.help
Subject: Re: text encoding of umlauts in latest view broken
Date: Sun, 25 Aug 2024 20:45:12 +0000
Organization: Rocksolid Light
Message-ID: <48ac10adc0f5726a56715951cf1e787d@www.novabbs.org>
References: <vaetmk$1q1gt$3@dont-email.me> <e2db830f8efe89dcea07f1bbb5ef6601$1@novabbs.org> <AABmyy240AwAAPD3.A3.flnews@t20.ybtra.de> <vafbki$1s7hg$1@dont-email.me> <AABmyzoMiikAARGb.A3.flnews@t20.ybtra.de> <cb45e76f63e84ef0c6588dc64c88a2fd$1@novabbs.org> <df91607f35e218a06711caea02ff4731@www.novabbs.org> <vag1sr$21ttr$1@dont-email.me>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Transfer-Encoding: 8bit
Injection-Info: i2pn2.org;
logging-data="3885802"; mail-complaints-to="usenet@i2pn2.org";
posting-account="gg+dDWHMzVrdxGO9Gmt8aqyeWDXqlxfqiuS0jX/WMXY";
User-Agent: Rocksolid Light
X-Face: .&YR-G(w(DZ$$,}%k=]*5*!p'=(anr"IT`wZG'2VWdfl\r)l[42u7JH`n(JUQ*e5*A|XCDf
?&\X&uwkl38"CYX3O8m}C8E4p'%N$2#kSTVzx{Ly|DjLT\Vk7NE}NQ(VC$Yq]i:7|z[.9iv^g>*8_B
H0=hZt'[%)4kG|
X-Rslight-Site: $2y$10$eOD0FOyme6FrReG6XRL7Q.gFbKPbYl9WzH2V8jLPybwu/VrhYej1W
X-Rslight-Posting-User: a93aefeeff923def71455caae2dbfb277a59e046
X-Spam-Checker-Version: SpamAssassin 4.0.0

View all headers

On Sun, 25 Aug 2024 19:49:46 +0000, Marco Moock wrote:

> On 25.08.2024 um 18:42 Uhr Retro Guy wrote:
>
>> Does this look better? https://postimg.cc/Zv4gwcVP
>
> Looks fine, umlauts are now working.

Great!

It turned out to be an issue specific to the headers for this poster,
and I'm sure some others that I haven't noticed. The character set was
not being properly extracted from the header due to the specific way the
header line was generated. This is now checked and the line is parsed
for this.

The fix will only work for newly spooled articles, as the "snippet" is
saved in the database for articles, it is not generated every time it is
displayed.

So new articles should display properly in Latest, but existing articles
will not change.

I have just pushed this fix, and several others, to devel.

--
Retro Guy

Path: eternal-september.org!news.eternal-september.org!.POSTED!not-for-mail
From: mm+usenet-es@dorfdsl.de (Marco Moock)
Newsgroups: rocksolid.nodes.help
Subject: Re: text encoding of umlauts in latest view broken
Date: Mon, 26 Aug 2024 13:02:02 +0200
Organization: A noiseless patient Spider
Lines: 13
Message-ID: <vahnba$2dce8$1@dont-email.me>
References: <vaetmk$1q1gt$3@dont-email.me>
<e2db830f8efe89dcea07f1bbb5ef6601$1@novabbs.org>
<AABmyy240AwAAPD3.A3.flnews@t20.ybtra.de>
<vafbki$1s7hg$1@dont-email.me>
<AABmyzoMiikAARGb.A3.flnews@t20.ybtra.de>
<cb45e76f63e84ef0c6588dc64c88a2fd$1@novabbs.org>
<df91607f35e218a06711caea02ff4731@www.novabbs.org>
<vag1sr$21ttr$1@dont-email.me>
<48ac10adc0f5726a56715951cf1e787d@www.novabbs.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=US-ASCII
Content-Transfer-Encoding: 7bit
Injection-Date: Mon, 26 Aug 2024 13:02:03 +0200 (CEST)
Injection-Info: dont-email.me; posting-host="cf83377bf555d1fd97063dfcabf5cb30";
logging-data="2535880"; mail-complaints-to="abuse@eternal-september.org"; posting-account="U2FsdGVkX197IxhszHDX+QXH+Kiq048v"
Cancel-Lock: sha1:doK1Qmwtu6hQ0uQ5GXZlQs0Plq4=

View all headers

On 25.08.2024 um 20:45 Uhr Retro Guy wrote:

> I have just pushed this fix, and several others, to devel.

Is this currently considered stable for production or should I wait
before switching to the current version?

--
kind regards
Marco

Send spam to 1724611512muell@cartoonies.org

Subject: Re: text encoding of umlauts in latest view broken
From: Retro Guy
Newsgroups: rocksolid.nodes.help
Organization: Rocksolid Light
Date: Mon, 26 Aug 2024 11:40 UTC
References: 1 2 3 4 5 6 7 8 9 10

Path: eternal-september.org!news.eternal-september.org!feeder3.eternal-september.org!i2pn.org!i2pn2.org!.POSTED!not-for-mail
From: retroguy@novabbs.com (Retro Guy)
Newsgroups: rocksolid.nodes.help
Subject: Re: text encoding of umlauts in latest view broken
Date: Mon, 26 Aug 2024 04:40:14 -0700
Organization: Rocksolid Light
Message-ID: <ebc199f5c4548f5619500b384117c651$1@novabbs.org>
References: <vaetmk$1q1gt$3@dont-email.me> <e2db830f8efe89dcea07f1bbb5ef6601$1@novabbs.org> <AABmyy240AwAAPD3.A3.flnews@t20.ybtra.de> <vafbki$1s7hg$1@dont-email.me> <AABmyzoMiikAARGb.A3.flnews@t20.ybtra.de> <cb45e76f63e84ef0c6588dc64c88a2fd$1@novabbs.org> <df91607f35e218a06711caea02ff4731@www.novabbs.org> <vag1sr$21ttr$1@dont-email.me> <48ac10adc0f5726a56715951cf1e787d@www.novabbs.org> <vahnba$2dce8$1@dont-email.me>
MIME-Version: 1.0
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: 7bit
Injection-Info: i2pn2.org;
logging-data="3963153"; mail-complaints-to="usenet@i2pn2.org";
posting-account="PGd4t4cXnWwgUWG9VtTiCsm47oOWbHLcTr4rYoM0Edo";
User-Agent: 40tude_Dialog/2.0.15.41
X-Spam-Checker-Version: SpamAssassin 4.0.0

View all headers

On Mon, 26 Aug 2024 13:02:02 +0200, Marco Moock wrote:

> On 25.08.2024 um 20:45 Uhr Retro Guy wrote:
>
>> I have just pushed this fix, and several others, to devel.
>
> Is this currently considered stable for production or should I wait
> before switching to the current version?

It should be fine. I just now merged devel into master, and pushed master
to gitlab.

So master is latest at the moment.

Please, always back up first.

You may wish to restart your nntp server(s) after upgrade, as
rslight-lib.php is modified. You'll need to kill them manually and let cron
restart them. In the upgraded version, restarting them is much simpler :)

Subject	Author
text encoding of umlauts in latest view broken	Marco Moock
Re: text encoding of umlauts in latest view broken	Retro Guy
Re: text encoding of umlauts in latest view broken	Marcel Logen
Re: text encoding of umlauts in latest view broken	Marco Moock
Re: text encoding of umlauts in latest view broken	Marcel Logen
Re: text encoding of umlauts in latest view broken	Retro Guy
Re: text encoding of umlauts in latest view broken	Retro Guy
Re: text encoding of umlauts in latest view broken	Marco Moock
Re: text encoding of umlauts in latest view broken	Retro Guy
Re: text encoding of umlauts in latest view broken	Marco Moock
Re: text encoding of umlauts in latest view broken	Retro Guy