Hello!

I notice that on my instance some groups are note being updated on the
system itself.

The articles itself are on i2pn2.org.

Some other groups are updated, but it is not limited to one non-working
group.

How can I further diagnose that?

--
kind regards
Marco

Send spam to 1731510775muell@stinkedores.dorfdsl.de

On Wed, 13 Nov 2024 15:21:48 +0000, Marco Moock wrote:

> Hello!
>
> I notice that on my instance some groups are note being updated on the
> system itself.
>
> The articles itself are on i2pn2.org.
>
> Some other groups are updated, but it is not limited to one non-working
> group.
>
> How can I further diagnose that?

Well that's not good :(

Can you check if the articles exist when connecting directly to the
rslight nntp server?

Are the groups that fail ALL groups below a certain line in groups.txt?

Can you grep a message-id of a failed article in spoolnews.log and see
if there is a log entry?

How is your disk space?

--
Retro Guy

On 13.11.2024 um 15:51 Uhr Retro Guy wrote:

> On Wed, 13 Nov 2024 15:21:48 +0000, Marco Moock wrote:
>
> > Hello!
> >
> > I notice that on my instance some groups are note being updated on
> > the system itself.
> >
> > The articles itself are on i2pn2.org.
> >
> > Some other groups are updated, but it is not limited to one
> > non-working group.
> >
> > How can I further diagnose that?
>
> Well that's not good :(
>
> Can you check if the articles exist when connecting directly to the
> rslight nntp server?

The internal server doesn't have the article.

> Are the groups that fail ALL groups below a certain line in
> groups.txt?

Looks a bit like, because the rocksolid stuff is working, but at least
2 groups in de.* are/were also working.

> Can you grep a message-id of a failed article in spoolnews.log and see
> if there is a log entry?

No logs for the missing article.
For other articles there are logs, so logging is working.
For today, I only see that the rocksolid and the NoCeM groups were
spooled.

> How is your disk space?

/dev/mmcblk1p2 14G 5,3G 8,1G 40% /

--
kind regards
Marco

Send spam to 1731509510muell@cartoonies.org

On Wed, 13 Nov 2024 20:28:15 +0100, Marco Moock wrote:

> On 13.11.2024 um 15:51 Uhr Retro Guy wrote:
>
>> On Wed, 13 Nov 2024 15:21:48 +0000, Marco Moock wrote:
>>
>>> Hello!
>>>
>>> I notice that on my instance some groups are note being updated on
>>> the system itself.
>>>
>>> The articles itself are on i2pn2.org.
>>>
>>> Some other groups are updated, but it is not limited to one
>>> non-working group.
>>>
>>> How can I further diagnose that?
>>
>> Well that's not good :(
>>
>> Can you check if the articles exist when connecting directly to the
>> rslight nntp server?
>
> The internal server doesn't have the article.
>
>> Are the groups that fail ALL groups below a certain line in
>> groups.txt?
>
> Looks a bit like, because the rocksolid stuff is working, but at least
> 2 groups in de.* are/were also working.

Can you check spoolnews.log and see if it stops spooling groups after the
last group that works. Maybe an error like can't connect to remote server
or such?

Once it gets to the groups that do not work, if it ever does, are there
errors in spoolnews about those groups?

I'm seeing a lot of:
<your server> can't read: Connection reset by peer

Which only started today at:
Nov 13 04:25:25

Any changes made at that time?

Maybe it's my end, but no such logs from my rslight servers, not that it
means it's not i2pn2 problem.

On Wed, 13 Nov 2024 13:42:49 -0700, Retro Guy wrote:

> On Wed, 13 Nov 2024 20:28:15 +0100, Marco Moock wrote:
>
>> On 13.11.2024 um 15:51 Uhr Retro Guy wrote:
>>
>>> On Wed, 13 Nov 2024 15:21:48 +0000, Marco Moock wrote:
>>>
>>>> Hello!
>>>>
>>>> I notice that on my instance some groups are note being updated on
>>>> the system itself.
>>>>
>>>> The articles itself are on i2pn2.org.
>>>>
>>>> Some other groups are updated, but it is not limited to one
>>>> non-working group.
>>>>
>>>> How can I further diagnose that?
>>>
>>> Well that's not good :(
>>>
>>> Can you check if the articles exist when connecting directly to the
>>> rslight nntp server?
>>
>> The internal server doesn't have the article.
>>
>>> Are the groups that fail ALL groups below a certain line in
>>> groups.txt?
>>
>> Looks a bit like, because the rocksolid stuff is working, but at least
>> 2 groups in de.* are/were also working.
>
> Can you check spoolnews.log and see if it stops spooling groups after the
> last group that works. Maybe an error like can't connect to remote server
> or such?
>
> Once it gets to the groups that do not work, if it ever does, are there
> errors in spoolnews about those groups?
>
> I'm seeing a lot of:
> <your server> can't read: Connection reset by peer
>
> Which only started today at:
> Nov 13 04:25:25
>
> Any changes made at that time?
>
> Maybe it's my end, but no such logs from my rslight servers, not that it
> means it's not i2pn2 problem.

Also, try i2pn2.org port 8991

On Wed, 13 Nov 2024 13:43:58 -0700, Retro Guy wrote:

> On Wed, 13 Nov 2024 13:42:49 -0700, Retro Guy wrote:
>
>> On Wed, 13 Nov 2024 20:28:15 +0100, Marco Moock wrote:
>>
>>> On 13.11.2024 um 15:51 Uhr Retro Guy wrote:
>>>
>>>> On Wed, 13 Nov 2024 15:21:48 +0000, Marco Moock wrote:
>>>>
>>>>> Hello!
>>>>>
>>>>> I notice that on my instance some groups are note being updated on
>>>>> the system itself.
>>>>>
>>>>> The articles itself are on i2pn2.org.
>>>>>
>>>>> Some other groups are updated, but it is not limited to one
>>>>> non-working group.
>>>>>
>>>>> How can I further diagnose that?
>>>>
>>>> Well that's not good :(
>>>>
>>>> Can you check if the articles exist when connecting directly to the
>>>> rslight nntp server?
>>>
>>> The internal server doesn't have the article.
>>>
>>>> Are the groups that fail ALL groups below a certain line in
>>>> groups.txt?
>>>
>>> Looks a bit like, because the rocksolid stuff is working, but at least
>>> 2 groups in de.* are/were also working.
>>
>> Can you check spoolnews.log and see if it stops spooling groups after the
>> last group that works. Maybe an error like can't connect to remote server
>> or such?
>>
>> Once it gets to the groups that do not work, if it ever does, are there
>> errors in spoolnews about those groups?
>>
>> I'm seeing a lot of:
>> <your server> can't read: Connection reset by peer
>>
>> Which only started today at:
>> Nov 13 04:25:25
>>
>> Any changes made at that time?
>>
>> Maybe it's my end, but no such logs from my rslight servers, not that it
>> means it's not i2pn2 problem.
>
> Also, try i2pn2.org port 8991

If you don't mind, could you email me some files please?

I'm looking for spoolnews.log, debug.log and, in <spooldir>/rocksolid there
should be a file <server>:<port>-remote_groups.dat

The last file will show the last article received for each group from the
server.

I did just realize that all my sites pull from i2pn2 via an internal
network, not i2pn2.org but internal ip addresses at the hosting site. Maybe
there's a packet loss issue at their (hosting) end? I don't really know
yet.

On Wed, 13 Nov 2024 15:17:59 -0700, Retro Guy wrote:

> On Wed, 13 Nov 2024 13:43:58 -0700, Retro Guy wrote:
>
>> On Wed, 13 Nov 2024 13:42:49 -0700, Retro Guy wrote:
>>
>>> On Wed, 13 Nov 2024 20:28:15 +0100, Marco Moock wrote:
>>>
>>>> On 13.11.2024 um 15:51 Uhr Retro Guy wrote:
>>>>
>>>>> On Wed, 13 Nov 2024 15:21:48 +0000, Marco Moock wrote:
>>>>>
>>>>>> Hello!
>>>>>>
>>>>>> I notice that on my instance some groups are note being updated on
>>>>>> the system itself.
>>>>>>
>>>>>> The articles itself are on i2pn2.org.
>>>>>>
>>>>>> Some other groups are updated, but it is not limited to one
>>>>>> non-working group.
>>>>>>
>>>>>> How can I further diagnose that?
>>>>>
>>>>> Well that's not good :(
>>>>>
>>>>> Can you check if the articles exist when connecting directly to the
>>>>> rslight nntp server?
>>>>
>>>> The internal server doesn't have the article.
>>>>
>>>>> Are the groups that fail ALL groups below a certain line in
>>>>> groups.txt?
>>>>
>>>> Looks a bit like, because the rocksolid stuff is working, but at least
>>>> 2 groups in de.* are/were also working.
>>>
>>> Can you check spoolnews.log and see if it stops spooling groups after the
>>> last group that works. Maybe an error like can't connect to remote server
>>> or such?
>>>
>>> Once it gets to the groups that do not work, if it ever does, are there
>>> errors in spoolnews about those groups?
>>>
>>> I'm seeing a lot of:
>>> <your server> can't read: Connection reset by peer
>>>
>>> Which only started today at:
>>> Nov 13 04:25:25
>>>
>>> Any changes made at that time?
>>>
>>> Maybe it's my end, but no such logs from my rslight servers, not that it
>>> means it's not i2pn2 problem.
>>
>> Also, try i2pn2.org port 8991
>
> If you don't mind, could you email me some files please?
>
> I'm looking for spoolnews.log, debug.log and, in <spooldir>/rocksolid there
> should be a file <server>:<port>-remote_groups.dat
>
> The last file will show the last article received for each group from the
> server.
>
> I did just realize that all my sites pull from i2pn2 via an internal
> network, not i2pn2.org but internal ip addresses at the hosting site. Maybe
> there's a packet loss issue at their (hosting) end? I don't really know
> yet.

I've modified spoolnews.php to try to isolate such issues, and add logging
to find exactly when/where it happens.

If we don't find a simple answer, I'll ask that you apply the new
spoolnews.php, which I will need to send you, to test further. It is not
yet in devel on gitlab, so no sense downloading what is there.

On Wed, 13 Nov 2024 22:47:45 +0000, Retro Guy wrote:

> On Wed, 13 Nov 2024 15:17:59 -0700, Retro Guy wrote:
>
>> On Wed, 13 Nov 2024 13:43:58 -0700, Retro Guy wrote:
>>
>>> On Wed, 13 Nov 2024 13:42:49 -0700, Retro Guy wrote:
>>>
>>>> On Wed, 13 Nov 2024 20:28:15 +0100, Marco Moock wrote:
>>>>
>>>>> On 13.11.2024 um 15:51 Uhr Retro Guy wrote:
>>>>>
>>>>>> On Wed, 13 Nov 2024 15:21:48 +0000, Marco Moock wrote:
>>>>>>
>>>>>>> Hello!
>>>>>>>
>>>>>>> I notice that on my instance some groups are note being updated on
>>>>>>> the system itself.
>>>>>>>
>>>>>>> The articles itself are on i2pn2.org.
>>>>>>>
>>>>>>> Some other groups are updated, but it is not limited to one
>>>>>>> non-working group.
>>>>>>>
>>>>>>> How can I further diagnose that?
>>>>>>
>>>>>> Well that's not good :(
>>>>>>
>>>>>> Can you check if the articles exist when connecting directly to the
>>>>>> rslight nntp server?
>>>>>
>>>>> The internal server doesn't have the article.
>>>>>
>>>>>> Are the groups that fail ALL groups below a certain line in
>>>>>> groups.txt?
>>>>>
>>>>> Looks a bit like, because the rocksolid stuff is working, but at least
>>>>> 2 groups in de.* are/were also working.
>>>>
>>>> Can you check spoolnews.log and see if it stops spooling groups after
>>>> the
>>>> last group that works. Maybe an error like can't connect to remote
>>>> server
>>>> or such?
>>>>
>>>> Once it gets to the groups that do not work, if it ever does, are there
>>>> errors in spoolnews about those groups?
>>>>
>>>> I'm seeing a lot of:
>>>> <your server> can't read: Connection reset by peer
>>>>
>>>> Which only started today at:
>>>> Nov 13 04:25:25
>>>>
>>>> Any changes made at that time?
>>>>
>>>> Maybe it's my end, but no such logs from my rslight servers, not that it
>>>> means it's not i2pn2 problem.
>>>
>>> Also, try i2pn2.org port 8991
>>
>> If you don't mind, could you email me some files please?
>>
>> I'm looking for spoolnews.log, debug.log and, in <spooldir>/rocksolid
>> there
>> should be a file <server>:<port>-remote_groups.dat
>>
>> The last file will show the last article received for each group from
>> the
>> server.
>>
>> I did just realize that all my sites pull from i2pn2 via an internal
>> network, not i2pn2.org but internal ip addresses at the hosting site.
>> Maybe
>> there's a packet loss issue at their (hosting) end? I don't really know
>> yet.
>
> I've modified spoolnews.php to try to isolate such issues, and add
> logging
> to find exactly when/where it happens.
>
> If we don't find a simple answer, I'll ask that you apply the new
> spoolnews.php, which I will need to send you, to test further. It is not
> yet in devel on gitlab, so no sense downloading what is there.

After discussion with some admins about this, and being able to
duplicate the issue myself, I believe it is now fixed.

It appeared to be an issue when polling groups that have no articles on
the remote server. Now, this is checked and the polling script moves on
instead of dies.

This change, and everything else from devel is now in master on gitlab:
https://gitlab.com/rslight-public/rocksolid-light

This version is live on all my sites now.

If there are further issues with this, we will address them asap.

--
Retro Guy

On 14.11.2024 um 15:12 Uhr Retro Guy wrote:

> This change, and everything else from devel is now in master on
> gitlab: https://gitlab.com/rslight-public/rocksolid-light
>
> This version is live on all my sites now.
>
> If there are further issues with this, we will address them asap.

Articles are now being downloaded and spooled, but in the web I get no
such group for all groups.
The group list is still there, but everything else isn't accessible or
gone.

--
kind regards
Marco

Send spam to 1731593546muell@cartoonies.org

Marco Moock wrote:

> On 14.11.2024 um 15:12 Uhr Retro Guy wrote:
>
>> This change, and everything else from devel is now in master on
>> gitlab: https://gitlab.com/rslight-public/rocksolid-light
>>
>> This version is live on all my sites now.
>>
>> If there are further issues with this, we will address them asap.
>
> Articles are now being downloaded and spooled, but in the web I get no
> such group for all groups.
> The group list is still there, but everything else isn't accessible or
> gone.
>

Please try restarting your nntp server:

touch /etc/rslight/nntp.reload

Then wait until cron.php runs again and see if that fixes it. It may be
simply an update to rslight-lib.php causing the issue.

--
Retro Guy

On 14.11.2024 um 16:28 Uhr Retro Guy wrote:

> Please try restarting your nntp server:
>
> touch /etc/rslight/nntp.reload
>
> Then wait until cron.php runs again and see if that fixes it. It may
> be simply an update to rslight-lib.php causing the issue.

I did that, but still nothing in the webserver.
How can I further investigate that?

--
kind regards
Marco

Send spam to 1731598107muell@cartoonies.org

On Thu, 14 Nov 2024 17:52:26 +0100
Marco Moock <mm+usenet-es@dorfdsl.de> wrote:

> On 14.11.2024 um 16:28 Uhr Retro Guy wrote:
>
> > Please try restarting your nntp server:
> >
> > touch /etc/rslight/nntp.reload
> >
> > Then wait until cron.php runs again and see if that fixes it. It may
> > be simply an update to rslight-lib.php causing the issue.
>
> I did that, but still nothing in the webserver.
> How can I further investigate that?

Wow, I really broke it!

I need an hour or so to fix this. I can duplicate the issue here on a test site.

I will be working on this right now!

--
Retro Guy <retroguy@i2pn2.org>

On Thu, 14 Nov 2024 10:05:56 -0700
Retro Guy <retroguy@i2pn2.org> wrote:

> On Thu, 14 Nov 2024 17:52:26 +0100
> Marco Moock <mm+usenet-es@dorfdsl.de> wrote:
>
> > On 14.11.2024 um 16:28 Uhr Retro Guy wrote:
> >
> > > Please try restarting your nntp server:
> > >
> > > touch /etc/rslight/nntp.reload
> > >
> > > Then wait until cron.php runs again and see if that fixes it. It may
> > > be simply an update to rslight-lib.php causing the issue.
> >
> > I did that, but still nothing in the webserver.
> > How can I further investigate that?
>
> Wow, I really broke it!
>
> I need an hour or so to fix this. I can duplicate the issue here on a test site.
>
> I will be working on this right now!

I found the problem. I caused it fixing the last problem :(

I have undone the problem, but need to "fix" the previous problem in a different way.

I'm on it.

--
Retro Guy <retroguy@i2pn2.org>

On Thu, 14 Nov 2024 10:05:56 -0700
Retro Guy <retroguy@i2pn2.org> wrote:

> On Thu, 14 Nov 2024 17:52:26 +0100
> Marco Moock <mm+usenet-es@dorfdsl.de> wrote:
>
> > On 14.11.2024 um 16:28 Uhr Retro Guy wrote:
> >
> > > Please try restarting your nntp server:
> > >
> > > touch /etc/rslight/nntp.reload
> > >
> > > Then wait until cron.php runs again and see if that fixes it. It may
> > > be simply an update to rslight-lib.php causing the issue.
> >
> > I did that, but still nothing in the webserver.
> > How can I further investigate that?
>
> Wow, I really broke it!
>
> I need an hour or so to fix this. I can duplicate the issue here on a test site.
>
> I will be working on this right now!

I have just emailed to you a new copy of newsportal.php and some instructions.

Please let me know what happens. It works on my test site (of course it does)

--
Retro Guy <retroguy@i2pn2.org>

On 14.11.2024 um 10:41 Uhr Retro Guy wrote:

> I have just emailed to you a new copy of newsportal.php and some
> instructions.

This email got rejected because the outgoing server of your provider is
listed in uceprotect level 1 (it sent enough spam to spamtraps).

I've now whitelisted the IP 104.245.209.246.
Please relay mail to me directly if possible to avoid that and tell it
the provider, so they cut off the spammer.

--
kind regards
Marco

Send spam to 1731577265muell@cartoonies.org

On Thu, 14 Nov 2024 10:05:56 -0700
Retro Guy <retroguy@i2pn2.org> wrote:

> On Thu, 14 Nov 2024 17:52:26 +0100
> Marco Moock <mm+usenet-es@dorfdsl.de> wrote:
>
> > On 14.11.2024 um 16:28 Uhr Retro Guy wrote:
> >
> > > Please try restarting your nntp server:
> > >
> > > touch /etc/rslight/nntp.reload
> > >
> > > Then wait until cron.php runs again and see if that fixes it. It may
> > > be simply an update to rslight-lib.php causing the issue.
> >
> > I did that, but still nothing in the webserver.
> > How can I further investigate that?
>
> Wow, I really broke it!
>
> I need an hour or so to fix this. I can duplicate the issue here on a test site.
>
> I will be working on this right now!

I need to test here ...

--
Retro Guy <retroguy@i2pn2.org>

On Thu, 14 Nov 2024 18:24:21 +0000, Marco Moock wrote:

> On 14.11.2024 um 10:41 Uhr Retro Guy wrote:
>
>> I have just emailed to you a new copy of newsportal.php and some
>> instructions.
>
> This email got rejected because the outgoing server of your provider is
> listed in uceprotect level 1 (it sent enough spam to spamtraps).
>
> I've now whitelisted the IP 104.245.209.246.
> Please relay mail to me directly if possible to avoid that and tell it
> the provider, so they cut off the spammer.

Ah, ok. I forgot to use my personal email to email to you, which is my
own mail server. I'll try to remember in the future.

Btw, the provider for email that is NOT my personal mail is Postmark.

--
Retro Guy

On Thu, 14 Nov 2024 19:01:11 +0000, Retro Guy wrote:

> On Thu, 14 Nov 2024 17:41:05 +0000, Retro Guy wrote:
>
>> On Thu, 14 Nov 2024 10:05:56 -0700
>> Retro Guy <retroguy@i2pn2.org> wrote:
>>
>>> On Thu, 14 Nov 2024 17:52:26 +0100
>>> Marco Moock <mm+usenet-es@dorfdsl.de> wrote:
>>>
>>>> On 14.11.2024 um 16:28 Uhr Retro Guy wrote:
>>>>
>>>> > Please try restarting your nntp server:
>>>> >
>>>> > touch /etc/rslight/nntp.reload
>>>> >
>>>> > Then wait until cron.php runs again and see if that fixes it. It may
>>>> > be simply an update to rslight-lib.php causing the issue.
>>>>
>>>> I did that, but still nothing in the webserver.
>>>> How can I further investigate that?
>>>
>>> Wow, I really broke it!
>>>
>>> I need an hour or so to fix this. I can duplicate the issue here on a
>>> test site.
>>>
>>> I will be working on this right now!
>>
>> I have just emailed to you a new copy of newsportal.php and some
>> instructions.
>>
>> Please let me know what happens. It works on my test site (of course it
>> does)
>
> I have pushed the change I emailed you about to master, along with a
> change to spoolnews.php.

Also, I (we) need to keep an eye that I did not re-introduce the
original bug (stop spooling after empty group)

--
Retro Guy

On Thu, 14 Nov 2024 17:41:05 +0000, Retro Guy wrote:

> On Thu, 14 Nov 2024 10:05:56 -0700
> Retro Guy <retroguy@i2pn2.org> wrote:
>
>> On Thu, 14 Nov 2024 17:52:26 +0100
>> Marco Moock <mm+usenet-es@dorfdsl.de> wrote:
>>
>>> On 14.11.2024 um 16:28 Uhr Retro Guy wrote:
>>>
>>> > Please try restarting your nntp server:
>>> >
>>> > touch /etc/rslight/nntp.reload
>>> >
>>> > Then wait until cron.php runs again and see if that fixes it. It may
>>> > be simply an update to rslight-lib.php causing the issue.
>>>
>>> I did that, but still nothing in the webserver.
>>> How can I further investigate that?
>>
>> Wow, I really broke it!
>>
>> I need an hour or so to fix this. I can duplicate the issue here on a
>> test site.
>>
>> I will be working on this right now!
>
> I have just emailed to you a new copy of newsportal.php and some
> instructions.
>
> Please let me know what happens. It works on my test site (of course it
> does)

I have pushed the change I emailed you about to master, along with a
change to spoolnews.php.

--
Retro Guy

On 14.11.2024 um 19:01 Uhr Retro Guy wrote:

> I have pushed the change I emailed you about to master, along with a
> change to spoolnews.php.

It fixed the missing groups.
Some were rebuild according to the message in the webserver, but now
work.

Although, I noticed that it didn't poll all articles from some threads,
e.g.
https://pi-dach.dorfdsl.de/rocksolid/article-flat.php?id=2472&group=de.comp.os.unix.linux.misc#2472
https://pi-dach.dorfdsl.de/rocksolid/article-flat.php?id=2481&group=de.comp.os.unix.linux.misc#2481

Some messages are still missing from the web interface at all (but are
on i2pn2.org) and are in the log:

/var/spool/rslight/log/debug.log:Nov 14 15:44:17 rocksolid Used Injection-Date 1731592628 for: <vh4vjk$k3tg$1@gwaiyur.mb-net.net>
/var/spool/rslight/log/spoolnews.log:Nov 14 15:44:17 rocksolid Spooling: de.comp.os.unix.linux.misc:76 <vh4vjk$k3tg$1@gwaiyur.mb-net.net>

<vh4vjk$k3tg$1@gwaiyur.mb-net.net>
<vguh8f$bm0$1@akk3-dmz.akk.uni-karlsruhe.de>

--
kind regards
Marco

Send spam to 1731607271muell@cartoonies.org

On Thu, 14 Nov 2024 19:48:37 +0000, Marco Moock wrote:

> On 14.11.2024 um 19:01 Uhr Retro Guy wrote:
>
>> I have pushed the change I emailed you about to master, along with a
>> change to spoolnews.php.
>
> It fixed the missing groups.
> Some were rebuild according to the message in the webserver, but now
> work.
>
> Although, I noticed that it didn't poll all articles from some threads,
> e.g.
> https://pi-dach.dorfdsl.de/rocksolid/article-flat.php?id=2472&group=de.comp.os.unix.linux.misc#2472
> https://pi-dach.dorfdsl.de/rocksolid/article-flat.php?id=2481&group=de.comp.os.unix.linux.misc#2481
>
> Some messages are still missing from the web interface at all (but are
> on i2pn2.org) and are in the log:
>
> /var/spool/rslight/log/debug.log:Nov 14 15:44:17 rocksolid Used
> Injection-Date 1731592628 for: <vh4vjk$k3tg$1@gwaiyur.mb-net.net>
> /var/spool/rslight/log/spoolnews.log:Nov 14 15:44:17 rocksolid Spooling:
> de.comp.os.unix.linux.misc:76 <vh4vjk$k3tg$1@gwaiyur.mb-net.net>
>
>
> <vh4vjk$k3tg$1@gwaiyur.mb-net.net>
> <vguh8f$bm0$1@akk3-dmz.akk.uni-karlsruhe.de>

Yes, I see the same issue also. I'm working on it but it might be the
case that we lose a few articles, which is unfortunate.

Please let it run for a while to allow thread_load_newsserver() to run
(via cron), which only runs a few times per hour. It MIGHT fix this, but
it may not also.

There is a way to force it, but it's time consuming. If you have a
critical group missing critical articles, you can rebuild the entire
group with (run as root or www user):

php /etc/rslight/scripts/maintenance -reset <group name>

This will erase ALL the articles and poll them again according to the
quantity you wish to go back from the server in overrides.inc.php:

'maxfirstrequest' => 100, (use a number here for how many articles to go
back from newest on server)

On Thu, 14 Nov 2024 20:00:18 +0000, Retro Guy wrote:

> On Thu, 14 Nov 2024 19:48:37 +0000, Marco Moock wrote:
>
>> On 14.11.2024 um 19:01 Uhr Retro Guy wrote:
>>
>>> I have pushed the change I emailed you about to master, along with a
>>> change to spoolnews.php.
>>
>> It fixed the missing groups.
>> Some were rebuild according to the message in the webserver, but now
>> work.
>>
>> Although, I noticed that it didn't poll all articles from some threads,
>> e.g.
>> https://pi-dach.dorfdsl.de/rocksolid/article-flat.php?id=2472&group=de.comp.os.unix.linux.misc#2472
>> https://pi-dach.dorfdsl.de/rocksolid/article-flat.php?id=2481&group=de.comp.os.unix.linux.misc#2481
>>
>> Some messages are still missing from the web interface at all (but are
>> on i2pn2.org) and are in the log:
>>
>> /var/spool/rslight/log/debug.log:Nov 14 15:44:17 rocksolid Used
>> Injection-Date 1731592628 for: <vh4vjk$k3tg$1@gwaiyur.mb-net.net>
>> /var/spool/rslight/log/spoolnews.log:Nov 14 15:44:17 rocksolid Spooling:
>> de.comp.os.unix.linux.misc:76 <vh4vjk$k3tg$1@gwaiyur.mb-net.net>
>>
>>
>> <vh4vjk$k3tg$1@gwaiyur.mb-net.net>
>> <vguh8f$bm0$1@akk3-dmz.akk.uni-karlsruhe.de>
>
> Yes, I see the same issue also. I'm working on it but it might be the
> case that we lose a few articles, which is unfortunate.
>
> Please let it run for a while to allow thread_load_newsserver() to run
> (via cron), which only runs a few times per hour. It MIGHT fix this, but
> it may not also.
>
> There is a way to force it, but it's time consuming. If you have a
> critical group missing critical articles, you can rebuild the entire
> group with (run as root or www user):
>
> php /etc/rslight/scripts/maintenance -reset <group name>
>
> This will erase ALL the articles and poll them again according to the
> quantity you wish to go back from the server in overrides.inc.php:
>
> 'maxfirstrequest' => 100, (use a number here for how many articles to go
> back from newest on server)

BTW, I just did the above on https://news.novabbs.org for
rocksolid.nodes.help and it did work.

If you happen to be using caching, restart memcacheD, or if using files,
clear out the disk cache directory, else it will continue to pull from
the cache.

I think I'll add a switch to maintenance.php to '-reset' all groups in
any given section, instead of needing to do them one at a time. Maybe
tomorrow.

--
Retro Guy

On Thu, 14 Nov 2024 22:56:10 +0000, Retro Guy wrote:

> On Thu, 14 Nov 2024 20:00:18 +0000, Retro Guy wrote:
>
>> On Thu, 14 Nov 2024 19:48:37 +0000, Marco Moock wrote:
>>
>>> On 14.11.2024 um 19:01 Uhr Retro Guy wrote:
>>>
>>>> I have pushed the change I emailed you about to master, along with a
>>>> change to spoolnews.php.
>>>
>>> It fixed the missing groups.
>>> Some were rebuild according to the message in the webserver, but now
>>> work.
>>>
>>> Although, I noticed that it didn't poll all articles from some threads,
>>> e.g.
>>> https://pi-dach.dorfdsl.de/rocksolid/article-flat.php?id=2472&group=de.comp.os.unix.linux.misc#2472
>>> https://pi-dach.dorfdsl.de/rocksolid/article-flat.php?id=2481&group=de.comp.os.unix.linux.misc#2481
>>>
>>> Some messages are still missing from the web interface at all (but are
>>> on i2pn2.org) and are in the log:
>>>
>>> /var/spool/rslight/log/debug.log:Nov 14 15:44:17 rocksolid Used
>>> Injection-Date 1731592628 for: <vh4vjk$k3tg$1@gwaiyur.mb-net.net>
>>> /var/spool/rslight/log/spoolnews.log:Nov 14 15:44:17 rocksolid Spooling:
>>> de.comp.os.unix.linux.misc:76 <vh4vjk$k3tg$1@gwaiyur.mb-net.net>
>>>
>>>
>>> <vh4vjk$k3tg$1@gwaiyur.mb-net.net>
>>> <vguh8f$bm0$1@akk3-dmz.akk.uni-karlsruhe.de>
>>
>> Yes, I see the same issue also. I'm working on it but it might be the
>> case that we lose a few articles, which is unfortunate.
>>
>> Please let it run for a while to allow thread_load_newsserver() to run
>> (via cron), which only runs a few times per hour. It MIGHT fix this, but
>> it may not also.
>>
>> There is a way to force it, but it's time consuming. If you have a
>> critical group missing critical articles, you can rebuild the entire
>> group with (run as root or www user):
>>
>> php /etc/rslight/scripts/maintenance -reset <group name>
>>
>> This will erase ALL the articles and poll them again according to the
>> quantity you wish to go back from the server in overrides.inc.php:
>>
>> 'maxfirstrequest' => 100, (use a number here for how many articles to go
>> back from newest on server)
>
> BTW, I just did the above on https://news.novabbs.org for
> rocksolid.nodes.help and it did work.
>
> If you happen to be using caching, restart memcacheD, or if using files,
> clear out the disk cache directory, else it will continue to pull from
> the cache.
>
> I think I'll add a switch to maintenance.php to '-reset' all groups in
> any given section, instead of needing to do them one at a time. Maybe
> tomorrow.

Ok, this is done. It's in devel on gitlab.

The only file that is needed to be updated is
<config_dir>/scripts/maintenance.php

It will reset ALL groups in an entire section. It will take quite a
while!

If you choose to use it:

1. Check your settings in <config_dir>/overrides.php:
// spoolnews.php
'maxarticles_per_run' => 100,
'maxfirstrequest' => 500,

'maxarticles_per_run' is how many articles to retrieve each run of
cron.php
'maxfirstrequest' is how far back to start, or how many articles to
subtract from the remote server's highest article number for the group
as a starting point.

2. Disable cron.php

3. Run php <config_dir>/scripts/maintenance.php -reset-section <section
name>

4. When it's eventually done, restart cron.php.

5. WAIT. It will take some time to retrieve the articles from the remote
server, and then it will take some more time for rslight to rebuild it's
number of articles for each group to display in the main page.

I did this on a test site and all went well.

BUT IT Will TAKE TIME.

--
Retro Guy

On Thu, 14 Nov 2024 23:47:28 +0000, Retro Guy wrote:

> On Thu, 14 Nov 2024 22:56:10 +0000, Retro Guy wrote:
>
>> On Thu, 14 Nov 2024 20:00:18 +0000, Retro Guy wrote:
>>
>>> On Thu, 14 Nov 2024 19:48:37 +0000, Marco Moock wrote:
>>>
>>>> On 14.11.2024 um 19:01 Uhr Retro Guy wrote:
>>>>
>>>>> I have pushed the change I emailed you about to master, along with a
>>>>> change to spoolnews.php.
>>>>
>>>> It fixed the missing groups.
>>>> Some were rebuild according to the message in the webserver, but now
>>>> work.
>>>>
>>>> Although, I noticed that it didn't poll all articles from some threads,
>>>> e.g.
>>>> https://pi-dach.dorfdsl.de/rocksolid/article-flat.php?id=2472&group=de.comp.os.unix.linux.misc#2472
>>>> https://pi-dach.dorfdsl.de/rocksolid/article-flat.php?id=2481&group=de.comp.os.unix.linux.misc#2481
>>>>
>>>> Some messages are still missing from the web interface at all (but are
>>>> on i2pn2.org) and are in the log:
>>>>
>>>> /var/spool/rslight/log/debug.log:Nov 14 15:44:17 rocksolid Used
>>>> Injection-Date 1731592628 for: <vh4vjk$k3tg$1@gwaiyur.mb-net.net>
>>>> /var/spool/rslight/log/spoolnews.log:Nov 14 15:44:17 rocksolid Spooling:
>>>> de.comp.os.unix.linux.misc:76 <vh4vjk$k3tg$1@gwaiyur.mb-net.net>
>>>>
>>>>
>>>> <vh4vjk$k3tg$1@gwaiyur.mb-net.net>
>>>> <vguh8f$bm0$1@akk3-dmz.akk.uni-karlsruhe.de>
>>>
>>> Yes, I see the same issue also. I'm working on it but it might be the
>>> case that we lose a few articles, which is unfortunate.
>>>
>>> Please let it run for a while to allow thread_load_newsserver() to run
>>> (via cron), which only runs a few times per hour. It MIGHT fix this, but
>>> it may not also.
>>>
>>> There is a way to force it, but it's time consuming. If you have a
>>> critical group missing critical articles, you can rebuild the entire
>>> group with (run as root or www user):
>>>
>>> php /etc/rslight/scripts/maintenance -reset <group name>
>>>
>>> This will erase ALL the articles and poll them again according to the
>>> quantity you wish to go back from the server in overrides.inc.php:
>>>
>>> 'maxfirstrequest' => 100, (use a number here for how many articles to go
>>> back from newest on server)
>>
>> BTW, I just did the above on https://news.novabbs.org for
>> rocksolid.nodes.help and it did work.
>>
>> If you happen to be using caching, restart memcacheD, or if using files,
>> clear out the disk cache directory, else it will continue to pull from
>> the cache.
>>
>> I think I'll add a switch to maintenance.php to '-reset' all groups in
>> any given section, instead of needing to do them one at a time. Maybe
>> tomorrow.
>
> Ok, this is done. It's in devel on gitlab.
>
> The only file that is needed to be updated is
> <config_dir>/scripts/maintenance.php
>
> It will reset ALL groups in an entire section. It will take quite a
> while!
>
> If you choose to use it:
>
> 1. Check your settings in <config_dir>/overrides.php:
> // spoolnews.php
> 'maxarticles_per_run' => 100,
> 'maxfirstrequest' => 500,
>
> 'maxarticles_per_run' is how many articles to retrieve each run of
> cron.php
> 'maxfirstrequest' is how far back to start, or how many articles to
> subtract from the remote server's highest article number for the group
> as a starting point.
>
> 2. Disable cron.php
>
> 3. Run php <config_dir>/scripts/maintenance.php -reset-section <section
> name>
>
> 4. When it's eventually done, restart cron.php.
>
> 5. WAIT. It will take some time to retrieve the articles from the remote
> server, and then it will take some more time for rslight to rebuild it's
> number of articles for each group to display in the main page.
>
> I did this on a test site and all went well.
>
> BUT IT Will TAKE TIME.

Probably a better way is with the release in devel, using the new
feature in maintenance.php:

-refill: Go back x articles and retrieve missing from remote server
(-refill alt.test 200)
This will go back x articles from the remote server's newest article
number, and pull articles that you do not already have. It will skip
duplicates.

Using that would require performing an upgrade of what's currently in
devel, soon to be in master, as the change requires more than just
replacing maintenance.php.

I myself have chosen on my sites to just leave them alone, and let there
be that day in 2024 where a few articles are missing. I'm not sure it's
worth all the effort to track them down and fill them in. Now, if a user
specifically requests a few articles in a particular group, I may do it.
Otherwise, it's all in the past to me.

But I will continue to add spool repair features to mainenance.php ast
time goes on. It has been a useful script to me in the past. One feature
I like is that if, for example, you have a group with 100,000 messages
on your site and want to provide it to me, you just send me the
<group>-articles.db3 file and I can import the group into my site.
That's probably the feature I have used the most between sites.

Anyway, I hope to soon forget this thread ever happened :)

--
Retro Guy

On 14.11.2024 um 20:00 Uhr Retro Guy wrote:

> There is a way to force it, but it's time consuming. If you have a
> critical group missing critical articles, you can rebuild the entire
> group with (run as root or www user):
>
> php /etc/rslight/scripts/maintenance -reset <group name>
>
> This will erase ALL the articles and poll them again according to the
> quantity you wish to go back from the server in overrides.inc.php:

PHP Fatal error: Uncaught PDOException: SQLSTATE[HY000]: General error: 5 database is locked in /var/www/html/rocksolid/newsportal.php:3202
Stack trace:
#0 /var/www/html/rocksolid/newsportal.php(3202): PDOStatement->execute()
#1 /etc/rslight/scripts/maintenance.php(293): delete_message()
#2 /etc/rslight/scripts/maintenance.php(74): remove_articles()
#3 {main}
thrown in /var/www/html/rocksolid/newsportal.php on line 3202

I've shut down the cronjob systemd stuff during the run.

Is there another way to remove the group entirely and the restart from
the beginning for that group?
Or even all articles/groups, without removing settings etc.?

--
kind regards
Marco

Send spam to 1731610818muell@cartoonies.org