Rocksolid Light

News from da outaworlds

mail  files  register  groups  login

Message-ID:  

Your heart is pure, and your mind clear, and your soul devout.

rocksolid / Rocksolid Nodes Announce / Re: Branch status for RSLight

SubjectAuthor
* Branch status for RSLightRetro Guy
`* Re: Branch status for RSLightRetro Guy
 `* Re: Branch status for RSLightvga256
  `* Re: Branch status for RSLightRetro Guy
   `* Re: Branch status for RSLightvga256
    `- Re: Branch status for RSLightRetro Guy

1
Subject: Branch status for RSLight
From: Retro Guy
Newsgroups: rocksolid.nodes.announce
Organization: Rocksolid Light
Date: Mon, 14 Aug 2023 13:34 UTC
Path: eternal-september.org!news.eternal-september.org!i2pn.org!i2pn2.org!.POSTED!not-for-mail
From: retro.guy@rocksolidbbs.com (Retro Guy)
Newsgroups: rocksolid.nodes.announce
Subject: Branch status for RSLight
Date: Mon, 14 Aug 2023 13:34:06 +0000
Organization: Rocksolid Light
Message-ID: <7836e57bd097594c8d0c4008d3179bb9@news.novabbs.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Transfer-Encoding: 8bit
Injection-Info: i2pn2.org;
logging-data="2472323"; mail-complaints-to="usenet@i2pn2.org";
posting-account="PGd4t4cXnWwgUWG9VtTiCsm47oOWbHLcTr4rYoM0Edo";
User-Agent: Rocksolid Light 0.9.0
X-Rslight-Site: $2y$10$2qZFDpfuQ0lVYMPyA5QOs.Tp8HYbx2Nt0P5HHZZTbGudwtIQZrkbu
X-Face: .&YR-G(w(DZ$$,}%k=]*5*!p'=(anr"IT`wZG'2VWdfl\r)l[42u7JH`n(JUQ*e5*A|XCDf
?&\X&uwkl38"CYX3O8m}C8E4p'%N$2#kSTVzx{Ly|DjLT\Vk7NE}NQ(VC$Yq]i:7|z[.9iv^g>*8_B
H0=hZt'[%)4kG|
X-Rslight-Posting-User: 91053d4a47d51b416144568e5a1040f05e31ed1b
X-Spam-Checker-Version: SpamAssassin 4.0.0 (2022-12-13) on i2pn2.org
View all headers

The 'messaging' branch has been merged into master.

There is a new branch, 'overview'. You don't want to pull from this branch atm as it contains major changes and will not work with the overview database previous to these changes.

Currently, https://news.novabbs.org is running with all these changes, and seems to be running well and substantially faster.

The changes are as follows:

Completely remove the use of the per group flat file overview databases, in favor of only the .db3 site overview. The .db3 was already used, so making it match the features of the flat file db made sense, plus, the .db3 is much faster.

Modified import-db3.php to bring 'articles-overview.db3' up to date (to work with this version). It requires a one-time run of import-db3.php to update the .db3, then you can update to this version (0.9.0).

Renamed the function rslight_db_open to overview_db_open throughout the code. It makes much more sense, and easier to avoid using the wrong function.

During the process of making these changes, several fixes for tradspool were made.

There is a release link on github.com for version 0.8.5, which contains all the code previous to these changes.

--
Retro Guy

Subject: Re: Branch status for RSLight
From: Retro Guy
Newsgroups: rocksolid.nodes.announce
Organization: Rocksolid Light
Date: Tue, 15 Aug 2023 13:57 UTC
References: 1
Path: eternal-september.org!news.eternal-september.org!i2pn.org!i2pn2.org!.POSTED!not-for-mail
From: retro.guy@rocksolidbbs.com (Retro Guy)
Newsgroups: rocksolid.nodes.announce
Subject: Re: Branch status for RSLight
Date: Tue, 15 Aug 2023 13:57:21 +0000
Organization: Rocksolid Light
Message-ID: <60a5a58fdaa965ca1d88bd7f72fa88d1@news.novabbs.org>
References: <7836e57bd097594c8d0c4008d3179bb9@news.novabbs.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Transfer-Encoding: 8bit
Injection-Info: i2pn2.org;
logging-data="2586748"; mail-complaints-to="usenet@i2pn2.org";
posting-account="PGd4t4cXnWwgUWG9VtTiCsm47oOWbHLcTr4rYoM0Edo";
User-Agent: Rocksolid Light 0.9.0
X-Rslight-Site: $2y$10$NRIw9wKDfnIb1pPFMM6E3eHD2jUdkLeO0x7ioRIvKmDE/y362aDv6
X-Spam-Checker-Version: SpamAssassin 4.0.0 (2022-12-13) on i2pn2.org
X-Rslight-Posting-User: 91053d4a47d51b416144568e5a1040f05e31ed1b
X-Face: .&YR-G(w(DZ$$,}%k=]*5*!p'=(anr"IT`wZG'2VWdfl\r)l[42u7JH`n(JUQ*e5*A|XCDf
?&\X&uwkl38"CYX3O8m}C8E4p'%N$2#kSTVzx{Ly|DjLT\Vk7NE}NQ(VC$Yq]i:7|z[.9iv^g>*8_B
H0=hZt'[%)4kG|
View all headers

Retro Guy wrote:

> Completely remove the use of the per group flat file overview databases, in favor of only the .db3 site overview. The .db3 was already used, so making it match the features of the flat file db made sense, plus, the .db3 is much faster.

This change seems substanially faster since we're not scrolling through a huge flat file over and over again. Just reading/writing from a sorted database.

--
Retro Guy

Subject: Re: Branch status for RSLight
From: vga256
Newsgroups: rocksolid.nodes.announce
Organization: Rocksolid Light
Date: Tue, 15 Aug 2023 13:59 UTC
References: 1 2
Path: eternal-september.org!news.eternal-september.org!i2pn.org!i2pn2.org!.POSTED!not-for-mail
From: vga@vga256.com (vga256)
Newsgroups: rocksolid.nodes.announce
Subject: Re: Branch status for RSLight
Date: Tue, 15 Aug 2023 13:59:24 +0000
Organization: Rocksolid Light
Message-ID: <f5586f20dcb16e14b336ce491167ab2b@news.novabbs.org>
References: <7836e57bd097594c8d0c4008d3179bb9@news.novabbs.org> <60a5a58fdaa965ca1d88bd7f72fa88d1@news.novabbs.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Transfer-Encoding: 8bit
Injection-Info: i2pn2.org;
logging-data="2586748"; mail-complaints-to="usenet@i2pn2.org";
posting-account="PGd4t4cXnWwgUWG9VtTiCsm47oOWbHLcTr4rYoM0Edo";
User-Agent: Rocksolid Light 0.9.0
X-Rslight-Posting-User: 9d50238f0c8e65588ce75c7d7eaad42d47312328
X-Spam-Checker-Version: SpamAssassin 4.0.0 (2022-12-13) on i2pn2.org
X-Rslight-Site: $2y$10$r7c8ZQa2M2RCpO75D/nAP.X1WzLhaqDmLAEOsec/Fw2Eec7TWWK0C
View all headers

Very cool. I had no idea it was so expensive reading the flatfile.

--
Posted on Rocksolid Light

Subject: Re: Branch status for RSLight
From: Retro Guy
Newsgroups: rocksolid.nodes.announce
Organization: Rocksolid Light
Date: Tue, 15 Aug 2023 14:33 UTC
References: 1 2 3
Path: eternal-september.org!news.eternal-september.org!i2pn.org!i2pn2.org!.POSTED!not-for-mail
From: retro.guy@rocksolidbbs.com (Retro Guy)
Newsgroups: rocksolid.nodes.announce
Subject: Re: Branch status for RSLight
Date: Tue, 15 Aug 2023 14:33:05 +0000
Organization: Rocksolid Light
Message-ID: <6d161fc5c1f4989eeba22f7de5ad4423@news.novabbs.org>
References: <7836e57bd097594c8d0c4008d3179bb9@news.novabbs.org> <60a5a58fdaa965ca1d88bd7f72fa88d1@news.novabbs.org> <f5586f20dcb16e14b336ce491167ab2b@news.novabbs.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Transfer-Encoding: 8bit
Injection-Info: i2pn2.org;
logging-data="2589811"; mail-complaints-to="usenet@i2pn2.org";
posting-account="PGd4t4cXnWwgUWG9VtTiCsm47oOWbHLcTr4rYoM0Edo";
User-Agent: Rocksolid Light 0.9.0
X-Spam-Checker-Version: SpamAssassin 4.0.0 (2022-12-13) on i2pn2.org
X-Face: .&YR-G(w(DZ$$,}%k=]*5*!p'=(anr"IT`wZG'2VWdfl\r)l[42u7JH`n(JUQ*e5*A|XCDf
?&\X&uwkl38"CYX3O8m}C8E4p'%N$2#kSTVzx{Ly|DjLT\Vk7NE}NQ(VC$Yq]i:7|z[.9iv^g>*8_B
H0=hZt'[%)4kG|
X-Rslight-Posting-User: 91053d4a47d51b416144568e5a1040f05e31ed1b
X-Rslight-Site: $2y$10$utkBEcIgCL38e6v0GgacW.0c9.o4.IXzCgQqBCsMQOXehayE1coru
View all headers

vga256 wrote:

> Very cool. I had no idea it was so expensive reading the flatfile.

Sorting in sqlite makes a big difference. Speed is greatly improved if you create an INDEX on each column you plan to search for things. We have always had these indices in the dbs, but the flat file overview was really slowing things down.

For example, www.novabbs.com (com, not org) currently contains 3,640,739 articles. That's 3,640,739 lines in the overview file. That's a lot to search through, or delete one line from. This site is not yet running the new code.

--
Retro Guy

Subject: Re: Branch status for RSLight
From: vga256
Newsgroups: rocksolid.nodes.announce
Organization: Rocksolid Light
Date: Wed, 16 Aug 2023 00:44 UTC
References: 1 2 3 4
Path: eternal-september.org!news.eternal-september.org!i2pn.org!i2pn2.org!.POSTED!not-for-mail
From: vga@vga256.com (vga256)
Newsgroups: rocksolid.nodes.announce
Subject: Re: Branch status for RSLight
Date: Wed, 16 Aug 2023 00:44:17 +0000
Organization: Rocksolid Light
Message-ID: <254c53e2c2e9d341868e4ebca1f2f4db@news.novabbs.org>
References: <7836e57bd097594c8d0c4008d3179bb9@news.novabbs.org> <60a5a58fdaa965ca1d88bd7f72fa88d1@news.novabbs.org> <f5586f20dcb16e14b336ce491167ab2b@news.novabbs.org> <6d161fc5c1f4989eeba22f7de5ad4423@news.novabbs.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Transfer-Encoding: 8bit
Injection-Info: i2pn2.org;
logging-data="2641852"; mail-complaints-to="usenet@i2pn2.org";
posting-account="PGd4t4cXnWwgUWG9VtTiCsm47oOWbHLcTr4rYoM0Edo";
User-Agent: Rocksolid Light 0.9.0
X-Spam-Checker-Version: SpamAssassin 4.0.0 (2022-12-13) on i2pn2.org
X-Rslight-Site: $2y$10$kk7KXtJSwBYLzkJoYRENXO.KPCyoKeFaHxTYoMsFfSmihiO0.jpV6
X-Rslight-Posting-User: 9d50238f0c8e65588ce75c7d7eaad42d47312328
View all headers

Yeow! one of those scale issues I never think about.

--
Posted on Rocksolid Light

Subject: Re: Branch status for RSLight
From: Retro Guy
Newsgroups: rocksolid.nodes.announce
Organization: Rocksolid Light
Date: Wed, 16 Aug 2023 16:45 UTC
References: 1 2 3 4 5
Path: eternal-september.org!news.eternal-september.org!i2pn.org!i2pn2.org!.POSTED!not-for-mail
From: retro.guy@rocksolidbbs.com (Retro Guy)
Newsgroups: rocksolid.nodes.announce
Subject: Re: Branch status for RSLight
Date: Wed, 16 Aug 2023 16:45:32 +0000
Organization: Rocksolid Light
Message-ID: <f55718d12903a1172bf2ecec47b49be6@news.novabbs.org>
References: <7836e57bd097594c8d0c4008d3179bb9@news.novabbs.org> <60a5a58fdaa965ca1d88bd7f72fa88d1@news.novabbs.org> <f5586f20dcb16e14b336ce491167ab2b@news.novabbs.org> <6d161fc5c1f4989eeba22f7de5ad4423@news.novabbs.org> <254c53e2c2e9d341868e4ebca1f2f4db@news.novabbs.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Transfer-Encoding: 8bit
Injection-Info: i2pn2.org;
logging-data="2717537"; mail-complaints-to="usenet@i2pn2.org";
posting-account="PGd4t4cXnWwgUWG9VtTiCsm47oOWbHLcTr4rYoM0Edo";
User-Agent: Rocksolid Light 0.9.0
X-Spam-Checker-Version: SpamAssassin 4.0.0 (2022-12-13) on i2pn2.org
X-Rslight-Site: $2y$10$CGcBZF3Udshu1A6bsIWYi.9WBqnu0oTd.r4RgEuZ5cB7rJkspK8iC
X-Rslight-Posting-User: 91053d4a47d51b416144568e5a1040f05e31ed1b
X-Face: .&YR-G(w(DZ$$,}%k=]*5*!p'=(anr"IT`wZG'2VWdfl\r)l[42u7JH`n(JUQ*e5*A|XCDf
?&\X&uwkl38"CYX3O8m}C8E4p'%N$2#kSTVzx{Ly|DjLT\Vk7NE}NQ(VC$Yq]i:7|z[.9iv^g>*8_B
H0=hZt'[%)4kG|
View all headers

vga256 wrote:

> Yeow! one of those scale issues I never think about.

It's a big file, lol :)

Branch 'overview' now contains the changes to remove the use of overview flat file, and also the changes to add history.db3 to track (and not reuse) deleted article numbers.

Note: DO NOT UPGRADE to this branch yet, as the old articles-overview.db3 is not structured to handle the new overview. A task must be run first to upgrade the database.

I'll post here after testing a new ways that don't take forever to run. My initial test was to just write it from each group.db3, but there are faster ways that I'm looking into right now.

--
Retro Guy

1

rocksolid light 0.9.8
clearnet tor