On Tue, 25 Jun 2024 10:38:28 +0200, Bonita Montero wrote:

> Am 25.06.2024 um 09:06 schrieb Lawrence D'Oliveiro:
>
>> I wrote a memory-hog app for Android once, and found that allocating
>> large amounts of memory space had very little impact on the system.
>> Then when I added code to actually write data into those allocated
>> pages, that’s when it really started to break into a sweat ...
>
> Then android is also doing overcommit.

It is running a Linux kernel, and that tends to be the default setup in
Linux.

On 25/06/2024 12:28, Bonita Montero wrote:
>
> The interesting part is that after doing the first realloc()
> the memory being freee isn't reused for the next malloc().

That's entirely implementation dependent.

Andy

Am 26.06.2024 um 13:15 schrieb Vir Campestris:
> On 25/06/2024 12:28, Bonita Montero wrote:
>>
>> The interesting part is that after doing the first realloc()
>> the memory being freee isn't reused for the next malloc().
>
> That's entirely implementation dependent.

Of course, but that's what I wanted to test.
I traced the allocations with strace and the memory after the single
byte step actually is really freed and not marked as discardable.

Keith Thompson <Keith.S.Thompson+u@gmail.com> writes:
> Suppose you do something like:
>
> some_type *p = malloc(BIG_VALUE);
> // ...
> p = realloc(p, SMALL_VALUE);
>
> ... If realloc() succeeds and *does* relocate the object, p
> still points to memory that has now been deallocated, and you don't have
> a pointer to the newly allocated memory.

Surely some mistake?

However, such self-assignments are bad for the reasons you state later;
verify, then update.

Phil
--
We are no longer hunters and nomads. No longer awed and frightened, as we have
gained some understanding of the world in which we live. As such, we can cast
aside childish remnants from the dawn of our civilization.
-- NotSanguine on SoylentNews, after Eugen Weber in /The Western Tradition/

On 6/25/24 03:02, Lawrence D'Oliveiro wrote:
> On Mon, 24 Jun 2024 02:55:39 -0700, Keith Thompson wrote:
....
>> Having said all that, if realloc fails (indicated by returning a null
>> pointer), you still have the original pointer to the object.
>
> In other words, it’s safe to ignore any error from that last shrinking
> realloc? That’s good enough for me. ;)

No, you misunderstand:

q = realloc(p, SMALL_VALUE);

Then if q is null, p still points at the originally allocated memory. If
q is not null, then it may point at newly allocated memory, and p has in
indeterminate value. You cannot go forward ignoring the possibility that
no new object was allocated, because if you do, you have no way of
knowing which of the two pointers you can safely dereference. You need,
at least,

if(q)
p = q;

then you can safely use p, regardless of whether realloc() allocated new
memory.

On 6/25/24 06:05, Keith Thompson wrote:
....
> Suppose you do something like:
>
> some_type *p = malloc(BIG_VALUE);
> // ...
> p = realloc(p, SMALL_VALUE);
>
> If the realloc() succeeds and doesn't relocate and copy the object,
> you're fine. If realloc() succeeds and *does* relocate the object, p
> still points to memory that has now been deallocated, and you don't have
> a pointer to the newly allocated memory. ...

? I believe that, in that case, p does point to the newly allocated memory.

Phil Carmody <pc+usenet@asdf.org> writes:
> Keith Thompson <Keith.S.Thompson+u@gmail.com> writes:
>> Suppose you do something like:
>>
>> some_type *p = malloc(BIG_VALUE);
>> // ...
>> p = realloc(p, SMALL_VALUE);
>>
>> ... If realloc() succeeds and *does* relocate the object, p
>> still points to memory that has now been deallocated, and you don't have
>> a pointer to the newly allocated memory.
>
> Surely some mistake?

Yes, and thanks for catching it.

If realloc() succeeds, p points to the newly allocated memory; you no
longer have a pointer to the old deallocated memory, but that's a good
thing.

I think I was describing the behavior of calling realloc(p, SMALL_VALUE)
without assigning the result to anything.

> However, such self-assignments are bad for the reasons you state later;
> verify, then update.

Right.

--
Keith Thompson (The_Other_Keith) Keith.S.Thompson+u@gmail.com
void Void(void) { Void(); } /* The recursive call of the void */

James Kuyper <jameskuyper@alumni.caltech.edu> writes:
> On 6/25/24 06:05, Keith Thompson wrote:
> ...
>> Suppose you do something like:
>>
>> some_type *p = malloc(BIG_VALUE);
>> // ...
>> p = realloc(p, SMALL_VALUE);
>>
>> If the realloc() succeeds and doesn't relocate and copy the object,
>> you're fine. If realloc() succeeds and *does* relocate the object, p
>> still points to memory that has now been deallocated, and you don't have
>> a pointer to the newly allocated memory. ...
>
> ? I believe that, in that case, p does point to the newly allocated memory.

Yes, my mistake.

--
Keith Thompson (The_Other_Keith) Keith.S.Thompson+u@gmail.com
void Void(void) { Void(); } /* The recursive call of the void */

On Tue, 18 Jun 2024 11:46:36 +0100, Malcolm McLean wrote:

> Here are some real stats on file sizes, in case anone is interested.
>
> Data set, / OS Log-normal median & mean, Arithmetic mean, 50% occupied
> by (< mean)
>
> whole data set, 9.0 KB, 730 KB, 1.5 MB < 5.4 KB
> Mac OS 8.0 KB, 533 KB, 1.4 MB < 4.9 KB
> Windows 11.5 KB, 1.0 MB, 1.7 MB < 8.3 KB
> GNU/Linux 10.8 KB, 1.7MB, 2.2 MB < 4.8 KB
>
> https://www.researchgate.net/publication/353066615_How_Big_Are_Peoples%27_Computer_Files_File_Size_Distributions_Among_User-managed_Collections

I don’t see any error bars. Without those, it hard to attach any
significance to the differences in figures.

On Mon, 17 Jun 2024 18:02:49 +0300, Anton Shepelev
<anton.txt@g{oogle}mail.com> wrote:

>[cross-posted to: ci.stat.math]
>
Anton,

The post being responded to was originally to comp.lang.c
which I don't subscribe to.

I have a question that I suppose reflects on my news source,
GigaNews, or else on my reader, Forte Agent.

Was this thread something posted 15 or 20 years ago?

I tried to call up the original post by clicking on the Message
ID when looking at headers; nothing comes up when Agent goes
online to look. The header shows multiple earlier messages;
none of them come up for me.

My clicking on Message ID works elsewhere. The logical and
simple explanation is that this is a thread old enough that
GigaNews does not have it.

I suppose that someone else might be able to tell me, if their
supplier goes back further or if GigaNews is somehow failing
to show me something that is recent.

--
Rich Ulrich

Rich Ulrich <rich.ulrich@comcast.net> writes:
> On Mon, 17 Jun 2024 18:02:49 +0300, Anton Shepelev
> <anton.txt@g{oogle}mail.com> wrote:
>
>>[cross-posted to: ci.stat.math]
>>
> Anton,
>
> The post being responded to was originally to comp.lang.c
> which I don't subscribe to.
>
> I have a question that I suppose reflects on my news source,
> GigaNews, or else on my reader, Forte Agent.
>
> Was this thread something posted 15 or 20 years ago?
>
> I tried to call up the original post by clicking on the Message
> ID when looking at headers; nothing comes up when Agent goes
> online to look. The header shows multiple earlier messages;
> none of them come up for me.
>
> My clicking on Message ID works elsewhere. The logical and
> simple explanation is that this is a thread old enough that
> GigaNews does not have it.
>
> I suppose that someone else might be able to tell me, if their
> supplier goes back further or if GigaNews is somehow failing
> to show me something that is recent.

The first article in this thread was posted to comp.lang.c by Janis
Papanagnou on 17 Jun 2024.

There were several followups on the same day. The diret parent of your
article was cross-posted to comp.lang.c and sci.stat.math by Anton
Shepelev (his was the first cross-posted article in the thread).

--
Keith Thompson (The_Other_Keith) Keith.S.Thompson+u@gmail.com
void Void(void) { Void(); } /* The recursive call of the void */

On 7/2/2024 12:51 AM, Rich Ulrich wrote:
> On Mon, 17 Jun 2024 18:02:49 +0300, Anton Shepelev
> <anton.txt@g{oogle}mail.com> wrote:
>
>> [cross-posted to: ci.stat.math]
>>
> Anton,
>
> The post being responded to was originally to comp.lang.c
> which I don't subscribe to.
>
> I have a question that I suppose reflects on my news source,
> GigaNews, or else on my reader, Forte Agent.
>
> Was this thread something posted 15 or 20 years ago?
>
> I tried to call up the original post by clicking on the Message
> ID when looking at headers; nothing comes up when Agent goes
> online to look. The header shows multiple earlier messages;
> none of them come up for me.
>
> My clicking on Message ID works elsewhere. The logical and
> simple explanation is that this is a thread old enough that
> GigaNews does not have it.
>
> I suppose that someone else might be able to tell me, if their
> supplier goes back further or if GigaNews is somehow failing
> to show me something that is recent.
>

MID: <v4ojs8$gvji$1@dont-email.me>

http://al.howardknight.net/

That gives this URL, as a copy of the message kicking off the thread.

http://al.howardknight.net/?STYPE=msgid&MSGI=%3Cv4ojs8%24gvji%241%40dont-email.me%3E

Some USENET News clients can work from the MID directly, but Thunderbird does not.
A bare MID does not work for everyone.

Paul

On 29/06/2024 01:14, Lawrence D'Oliveiro wrote:
> On Tue, 18 Jun 2024 11:46:36 +0100, Malcolm McLean wrote:
>
>> Here are some real stats on file sizes, in case anone is interested.
>>
>> Data set, / OS Log-normal median & mean, Arithmetic mean, 50% occupied
>> by (< mean)
>>
>> whole data set, 9.0 KB, 730 KB, 1.5 MB < 5.4 KB
>> Mac OS 8.0 KB, 533 KB, 1.4 MB < 4.9 KB
>> Windows 11.5 KB, 1.0 MB, 1.7 MB < 8.3 KB
>> GNU/Linux 10.8 KB, 1.7MB, 2.2 MB < 4.8 KB
>>
>> https://www.researchgate.net/publication/353066615_How_Big_Are_Peoples%27_Computer_Files_File_Size_Distributions_Among_User-managed_Collections
>
> I don’t see any error bars. Without those, it hard to attach any
> significance to the differences in figures.
>
You don't need error bars becuase those fugures indicate a distribution.
The file are log-normally distributed wth given means and median. So the
spread is part of that data.

(Strictly you do need N to assess the significance, but it's got to be
one in a fazilion, the null that there is no difference at all in file
sizes between the platforms doesn't have a prayer. Unless you argue that
because the files came from one machine each, N is actualy 1).

--
Check out my hobby project.
http://malcolmmclean.github.io/babyxrc

Malcolm McLean <malcolm.arthur.mclean@gmail.com> writes:

> On 29/06/2024 01:14, Lawrence D'Oliveiro wrote:
>> On Tue, 18 Jun 2024 11:46:36 +0100, Malcolm McLean wrote:
>>
>>> Here are some real stats on file sizes, in case anone is interested.
>>>
>>> Data set, / OS Log-normal median & mean, Arithmetic mean, 50% occupied
>>> by (< mean)
>>>
>>> whole data set, 9.0 KB, 730 KB, 1.5 MB < 5.4 KB
>>> Mac OS 8.0 KB, 533 KB, 1.4 MB < 4.9 KB
>>> Windows 11.5 KB, 1.0 MB, 1.7 MB < 8.3 KB
>>> GNU/Linux 10.8 KB, 1.7MB, 2.2 MB < 4.8 KB
>>>
>>> https://www.researchgate.net/publication/353066615_How_Big_Are_Peoples%27_Computer_Files_File_Size_Distributions_Among_User-managed_Collections
>> I don’t see any error bars. Without those, it hard to attach any
>> significance to the differences in figures.
>>
> You don't need error bars becuase those fugures indicate a
> distribution. The file are log-normally distributed wth given means
> and median. So the spread is part of that data.

There are (or should be) two different distributions. Error bars are
intended to show the spread within the data. The log normal
distribution is across the data.

Now I suspect they didn't do it this way and just amalgamated all the
file save data into one, but that explains rather than excuses the lack
of error bars!

--
Ben.

On Mon, 01 Jul 2024 22:10:00 -0700, Keith Thompson
<Keith.S.Thompson+u@gmail.com> wrote:

>Rich Ulrich <rich.ulrich@comcast.net> writes:
>> On Mon, 17 Jun 2024 18:02:49 +0300, Anton Shepelev
>> <anton.txt@g{oogle}mail.com> wrote:
>>
>>>[cross-posted to: ci.stat.math]
>>>
>> Anton,
>>
>> The post being responded to was originally to comp.lang.c
>> which I don't subscribe to.
>>
>> I have a question that I suppose reflects on my news source,
>> GigaNews, or else on my reader, Forte Agent.
>>
>> Was this thread something posted 15 or 20 years ago?
>>
>> I tried to call up the original post by clicking on the Message
>> ID when looking at headers; nothing comes up when Agent goes
>> online to look. The header shows multiple earlier messages;
>> none of them come up for me.
>>
>> My clicking on Message ID works elsewhere. The logical and
>> simple explanation is that this is a thread old enough that
>> GigaNews does not have it.
>>
>> I suppose that someone else might be able to tell me, if their
>> supplier goes back further or if GigaNews is somehow failing
>> to show me something that is recent.
>
>The first article in this thread was posted to comp.lang.c by Janis
>Papanagnou on 17 Jun 2024.
>
>There were several followups on the same day. The diret parent of your
>article was cross-posted to comp.lang.c and sci.stat.math by Anton
>Shepelev (his was the first cross-posted article in the thread).

Thanks, so it looks like a failure by GigaNews to retrieve the
recent posts.

I did see a bunch of cross-posted followups. I don't know C,
and I thought there could be more context.

Looking at original, absent posts is something I've done dozens
of times over the years. Never a problem, except fora time or two
with posts from the 1990s. I've used GigaNews since my regular
ISP stopped providing Usenet access, maybe 15 years ago.

--
Rich Ulrich

On Tue, 2 Jul 2024 03:02:07 -0400, Paul <nospam@needed.invalid> wrote:

>On 7/2/2024 12:51 AM, Rich Ulrich wrote:
>> On Mon, 17 Jun 2024 18:02:49 +0300, Anton Shepelev
>> <anton.txt@g{oogle}mail.com> wrote:
>>
>>> [cross-posted to: ci.stat.math]
>>>
>> Anton,
>>
>> The post being responded to was originally to comp.lang.c
>> which I don't subscribe to.
>>
>> I have a question that I suppose reflects on my news source,
>> GigaNews, or else on my reader, Forte Agent.
>>
>> Was this thread something posted 15 or 20 years ago?
>>
>> I tried to call up the original post by clicking on the Message
>> ID when looking at headers; nothing comes up when Agent goes
>> online to look. The header shows multiple earlier messages;
>> none of them come up for me.
>>
>> My clicking on Message ID works elsewhere. The logical and
>> simple explanation is that this is a thread old enough that
>> GigaNews does not have it.
>>
>> I suppose that someone else might be able to tell me, if their
>> supplier goes back further or if GigaNews is somehow failing
>> to show me something that is recent.
>>
>
>MID: <v4ojs8$gvji$1@dont-email.me>
>
>http://al.howardknight.net/
>
>That gives this URL, as a copy of the message kicking off the thread.
>
> http://al.howardknight.net/?STYPE=msgid&MSGI=%3Cv4ojs8%24gvji%241%40dont-email.me%3E
>

Yes, that's the message I see when I plug the Message ID
into the program at http://al.howardknight.net/

Thanks. I'm saving that.

>Some USENET News clients can work from the MID directly, but Thunderbird does not.
>A bare MID does not work for everyone.

Forte Agent invites me to click on the MID; asks if it is a
mail or MID; asks if it should search the net. It still works
when I test it on an old message in another group.

--
Rich Ulrich

On Tue, 02 Jul 2024 11:52:56 -0400, Rich Ulrich
<rich.ulrich@comcast.net> wrote:

>
>Forte Agent invites me to click on the MID; asks if it is a
>mail or MID; asks if it should search the net. It still works
>when I test it on an old message in another group.
>

Now it occurs to me -- It actually does make sense,
economically, if what is searched online is limited the
groups I subscribe to, or (even) only the group that
is currently active.

--
Rich Ulrich

On 7/2/2024 11:58 AM, Rich Ulrich wrote:
> On Tue, 02 Jul 2024 11:52:56 -0400, Rich Ulrich
> <rich.ulrich@comcast.net> wrote:
>
>>
>> Forte Agent invites me to click on the MID; asks if it is a
>> mail or MID; asks if it should search the net. It still works
>> when I test it on an old message in another group.
>>
>
> Now it occurs to me -- It actually does make sense,
> economically, if what is searched online is limited the
> groups I subscribe to, or (even) only the group that
> is currently active.
>

Every device has "retention", but retention is limited.

Whether it's a search site, or a USENET server (even Forte had
their own news server, at one time), you need retention for
older articles to be search-able either as body text, or as
a <mid>.

Now that Google Groups is no longer connected to USENET,
that's one fewer places with a decent-sized archive. Google closed
their service, after the ThaiSpam incident. The Eternal-September
server, changed from one server to a two-server setup. The
Transit Server had Spam Assassin loaded on it, removing THaiSpam,
and the second server continued to offer normal ("filtered") service.

The comp.lang.c group was one of the groups under attack. Since
Google was letting the spam in, now that Google is disconnected,
the spam is gone, and the readership on CLC has gone up.

Paul

On 7/2/24 15:09, Paul wrote:
....
> Now that Google Groups is no longer connected to USENET,

I just checked, and as I had expected, Google is still connected.

> that's one fewer places with a decent-sized archive. Google closed
> their service, after the ThaiSpam incident.

They didn't close their service. They just stopped adding new messages
to their archives. The messages that were stored prior to the closing
are still available for searching.

On 7/2/24 15:09, Paul wrote:
....
> Now that Google Groups is no longer connected to USENET,
....
> that's one fewer places with a decent-sized archive. Google closed
> their service, after the ThaiSpam incident.

While they no longer store new messages, they still have one of the
largest archives of old messages, and it's still available for searching.

That’s an assumption of the parametric fit, not a fact of the data. Error
bars would indicate how close the fit is.

I had plumb forgot about this solution of mine:

> p0 = 1-e^(L*x0) ,
> p1 = 1-e^(L*x1) ,
> x1 = k*x0 (by our strategy), =>
> p1 = 1-(1-p0)^k .
>
> which does not depend on the distribution and lets us
> generalise this approach for any distribution:
>
> x1 = Q( 1 - ( 1 - CDF(x0) )^k )
> where:
>
> x0 : the required size
> x1 : the new recommended capacity
> Q(p) : the p-Quantile of the given distribution
> CDF(x): the CDF of the given distribution
> k>1 : balance between speed and space efficiency

Let us test it with the exponential distribution, for which:

Q (p) = -Ln( 1 - p )/L
CDF(x) = 1 - e^(-Lx)

Substituting these into the equation for x1:

x1 = Q ( 1 - ( 1 - ( 1 - e^(-Lx0) ) )^k ) =
Q ( 1 - ( e^(-Lx0) )^k ) =
Q ( 1 - e^(-kLx0) ) =
-Ln( e^(-kLx0) )/L = k*x0 (QED)

That is, my solution is a/the generalisation of the
exponential growth strategy.

--
() ascii ribbon campaign -- against html e-mail
/\ www.asciiribbon.org -- against proprietary attachments

Rich Ulrich:

> Thanks, so it looks like a failure by GigaNews to retrieve
> the recent posts.
>
> I did see a bunch of cross-posted followups. I don't know
> C, and I thought there could be more context.

Characters are being read in (say, from a file) sequentially
and stored in computer memory (RAM) in an "array" -- a
linear data structure storing elements (our characters) in a
sequential order, one ofter the other at addresses with
increasing indexes -- somewhat like a mathematical vector.

In order to store a character in an array, sufficient memory
has to be "allocated" for it, but while reading we do not
know beforehand the size of the file (or the total length of
the sequence), and therefore increase the allocated aray
size prospectively once the previous allocation is filled.
This operaion is called `realloc' and frequently involves
the tedious copying of the entire array onto a new location
in memory, taking a time in proportion to the number
elemennts so far allocated.

The question is to develop an optimal allcation strategy for
a given distribution of file sizes. The fasted solution is
to allocate a gigantic array beforehand, but it is a
terrible waste of memory. The slowet solution is to
reallcoate for each single character read it, but is a
terrible waste of CPU time. As I understand the problem, a
strategy is needed that manifests some compromise between
the extremes.

> Looking at original, absent posts is something I've done
> dozens of times over the years. Never a problem, except
> fora time or two with posts from the 1990s. I've used
> GigaNews since my regular ISP stopped providing Usenet
> access, maybe 15 years ago.

Just in case, there are many totally free Usenet servers,
e.g.:
http://www.eternal-september.org/
https://www.i2pn2.org/

and even a web interface:

https://www.novabbs.com

--
() ascii ribbon campaign -- against html e-mail
/\ www.asciiribbon.org -- against proprietary attachments

On Mon, 8 Jul 2024 20:01:21 +0300, Anton Shepelev
<anton.txt@g{oogle}mail.com> wrote:

>Rich Ulrich:
>
>> Thanks, so it looks like a failure by GigaNews to retrieve
>> the recent posts.
>>
>> I did see a bunch of cross-posted followups. I don't know
>> C, and I thought there could be more context.

<snip. Thanks for the details.>

Okay, today I discovered that there was no failure by Giganews
or by Forte Agent -- Instead, there was BEHAVIOR by Agent
that I was not aware of.

Today, was looking at All Desks (Agent terminology) to see what
was in Sent, and I noticed there were messages in Inbox -- Those
were the messages that I thought Agent had failed to retrieve.
(Nice discussion there.)

I guess - every other time I've clicked on Message-ID, the old
message was in the group where I was reading and the old one
showed up where I was reading. I've read the Agent group for
ages, and I don't remember this feature ever being mentioned.

Live and learn.

--
Rich Ulrich

Rich Ulrich:

> Today, was looking at All Desks (Agent terminology) to see
> what was in Sent, and I noticed there were messages in
> Inbox -- Those were the messages that I thought Agent had
> failed to retrieve. (Nice discussion there.)

Looking forward to your take on the problem. Mine is rather
simplisitc, but can be easily tested with several
distributions. This is like extrapolation: we know the
optimal solution for the given distribution (exponential)
and want to devise a general method to get the optimal
solution for any given distribution.

Malcolm, if you are still interested, can you provide a test
program that measures some statiscits for various allocation
strategies on various distributions, inclusing the
exponential?

--
() ascii ribbon campaign -- against html e-mail
/\ www.asciiribbon.org -- against proprietary attachments