John Ames <commodorejohn@gmail.com> writes:
> Rich <rich@example.invalid> wrote:
>> Yes, our local nymshift troll seems to clearly not know what a pipe
>> is, nor care to learn either.
>
> I *think* what he's meaning to say is this: while you can transfer any
> arbitrary amount of data *through* a pipe, there is an upper limit to
> how much you can have *in* a pipe at any one time; eventually, you hit
> either *A.* an OS-imposed limit on buffer size, at which point things
> start blocking as already discussed, or *B.* the upper bounds of
> system memory, at which point the system will either start swapping
> (in which case you lose any speed advantage) or blocking (as with
> limited buffer size.)

That’s a very charitable interpretation, particularly given that the
reality of how pipes work has been described two or three times
recently.

> That said, what probably shouldn't need saying here is that if you're
> filling up all available space in a pipe such that you're regularly
> hitting these limits, you're probably doing pipes wrong.

Pipes have a capacity of, typically, a few kilobytes and when that
capacity is reached, writers block until readers drain them.

All this stuff about filling RAM or swapping is just nonsense.

--
https://www.greenend.org.uk/rjk/

On 2024-12-18, Rich <rich@example.invalid> wrote:
> Robert Riches <spamtrap42@jacob21819.net> wrote:
>> On 2024-12-18, Rich <rich@example.invalid> wrote:
>>> 186282@ud0s4.net <186283@ud0s4.net> wrote:
>>>> On 12/17/24 8:23 PM, Lawrence D'Oliveiro wrote:
>>>>> On Tue, 17 Dec 2024 13:34:30 +0000, Geoff Clare wrote:
>>>>>
>>>>>> With a pipe or FIFO, you just use simple read and write operations
>>>>>> and the system handles all the messy stuff for you. If the pipe
>>>>>> reaches capacity, write blocks until there is room to write some
>>>>>> more; if the pipe becomes empty, read blocks until there is more
>>>>>> data available; when read returns EOF that's the end of the data.
>>>>>
>>>>> Yup. Furthermore:
>>>>>
>>>>> * When the last writer closes its end, any remaining read attempts
>>>>> get EOF.
>>>>> * When the last reader closes its end, any remaining write attempts
>>>>> get “broken pipe”.
>>>>
>>>> But you're still limited to the amount of RAM the system can
>>>> access.
>>>
>>> Not with a pipe or FIFO, which is what is being discussed above.
>>>
>>> The amount of data you can transfer over a pipe is not in any way
>>> limited by system memory size or any other system imposed limits.
>>>
>>>> These days that's probably a LOT - but might NOT be,
>>>> esp for 'embedded' type boards like the older PIs,
>>>> BBBs and such. Never assume the user has essentially
>>>> infinite RAM.
>>>
>>> The system will not have infinite RAM. You can transfer infinite data
>>> over a pipe (although it will take a while to reach infinity).
>>
>> A pipe is _NOT_ limited to system RAM!
>>
>> Using a named pipe on a Raspberry Pi model 1 with a _half_ GB of
>> total RAM, I would routinely transfer _several_ GB in a single
>> stream from an mplayer process to a netcat process. The only
>> reason that's not currently happening every night these days is
>> the amplified TV antenna lost too much gain due to age, attic
>> heat, etc.
>
> While you are correct, you responded to the wrong post. I pointed out
> to the nymshift troll the exact statement you made to me.

Yes, I was supporting your position with a concrete existence
proof. Not every reply needs to be a contradiction of the
immediately previous post.

I'm not certain, but I think I might have killfiled the nymshift
troll, so your post was the only one for which I had a reference
in order to contradict said nymshift troll.

--
Robert Riches
spamtrap42@jacob21819.net
(Yes, that is one of my email addresses.)

Robert Riches <spamtrap42@jacob21819.net> wrote:
> I'm not certain, but I think I might have killfiled the nymshift
> troll, so your post was the only one for which I had a reference in
> order to contradict said nymshift troll.

Ah, ok, now I get why you replied to my post. No worries in that case.

On Thu, 19 Dec 2024, Rich wrote:

> Robert Riches <spamtrap42@jacob21819.net> wrote:
>> I'm not certain, but I think I might have killfiled the nymshift
>> troll, so your post was the only one for which I had a reference in
>> order to contradict said nymshift troll.
>
> Ah, ok, now I get why you replied to my post. No worries in that case.
>

Harmony and peace have been restored!

On 11/24/24 2:14 PM, Richard Kettlewell wrote:
> Lew Pitcher <lew.pitcher@digitalfreehold.ca> writes:
>> On Sun, 24 Nov 2024 14:25:10 +0000, Rich wrote:
>>> 186282@ud0s4.net <186283@ud0s4.net> wrote:
>>>> On 11/23/24 4:25 PM, Lawrence D'Oliveiro wrote:
>>>>> That’s irrelevant. Pipes originated on the earliest Unix machine,
>>>>> which was a PDP-11 with only a 64kiB address space. They work great
>>>>> for pumping around gigabytes of data, but you don’t need
>>>>> gigabyte-sized memory buffers to do that.
>>>>
>>>> It all has to be SOMEWHERE ... if not in RAM then
>>>> on a mass storage device.
>>>
>>> Nope, at least not with pipes.
>>
>> Hold on a sec.... pipes are /buffered/ in RAM, so there's at least
>> a small bit of ram set aside for each open pipe.
>
> The word ‘all’ isn’t just decoration. The claim was ‘it all has to be
> somewhere’, and Rich’s point (as I understand it) is that it does not
> all have to be somewhere.
>
> For example,
>
> head -c $((1024*1024*1024)) /dev/urandom | sha256sum
>
> puts a gigabyte of data through a pipe, but at no point does anything
> allocate anywhere close to a gigabyte of storage of any kind.

There's truth here ... the gigabyte gets kinda 'spread out'.

But what if it was ten GB, or 100 ?

So, if you wanna count on sending vast quantities of
data in pipes, go ahead - it'll almost always work.

Until vids go 16k or something ...

As said LONG back ... each developer has to make
decisions based on their app and the 'likely'
systems/usage. There isn't really a 'wrong' or
'right' way here - just 'alternatives'. I like
terse pipes, you may not care.

"186282@ud0s4.net" <186283@ud0s4.net> writes:
> On 11/24/24 2:14 PM, Richard Kettlewell wrote:
>> Lew Pitcher <lew.pitcher@digitalfreehold.ca> writes:
>>> On Sun, 24 Nov 2024 14:25:10 +0000, Rich wrote:
>>>> 186282@ud0s4.net <186283@ud0s4.net> wrote:
>>>>> On 11/23/24 4:25 PM, Lawrence D'Oliveiro wrote:
>>>>>> That’s irrelevant. Pipes originated on the earliest Unix machine,
>>>>>> which was a PDP-11 with only a 64kiB address space. They work great
>>>>>> for pumping around gigabytes of data, but you don’t need
>>>>>> gigabyte-sized memory buffers to do that.
>>>>>
>>>>> It all has to be SOMEWHERE ... if not in RAM then
>>>>> on a mass storage device.
>>>>
>>>> Nope, at least not with pipes.
>>>
>>> Hold on a sec.... pipes are /buffered/ in RAM, so there's at least
>>> a small bit of ram set aside for each open pipe.
>>
>> The word ‘all’ isn’t just decoration. The claim was ‘it all has to
>> be somewhere’, and Rich’s point (as I understand it) is that it does
>> not all have to be somewhere.
>>
>> For example,
>> head -c $((1024*1024*1024)) /dev/urandom | sha256sum
>> puts a gigabyte of data through a pipe, but at no point does anything
>> allocate anywhere close to a gigabyte of storage of any kind.
>
> There's truth here ... the gigabyte gets kinda 'spread out'.
>
> But what if it was ten GB, or 100 ?

What difference do you think it’ll make? It’ll take proportionately
longer, but that’s all.

(SHA256 maxes out at a couple of exabytes, but that’s nothing to do with
pipes. Replace with a SHA3 variant if you want to think about sizes
beyond that.)

> So, if you wanna count on sending vast quantities of
> data in pipes, go ahead - it'll almost always work.

When do you think it will stop working?

--
https://www.greenend.org.uk/rjk/

On Wed, 25 Dec 2024 03:23:42 -0500, 186282@ud0s4.net wrote:

> But what if it was ten GB, or 100 ?

We get it, you have some kind of fear of pipes. But why? Were you scared
by a pipe as a baby? Dropped on your head and landed on one? Tried to use
them on Windows?

Just trying to understand the basis behind this phobia. It can’t simply
because you’re trying to save face, could it?

186282@ud0s4.net <186283@ud0s4.net> wrote:
> On 11/24/24 2:14 PM, Richard Kettlewell wrote:
>> Lew Pitcher <lew.pitcher@digitalfreehold.ca> writes:
>>> On Sun, 24 Nov 2024 14:25:10 +0000, Rich wrote:
>>>> 186282@ud0s4.net <186283@ud0s4.net> wrote:
>>>>> On 11/23/24 4:25 PM, Lawrence D'Oliveiro wrote:
>>>>>> That???s irrelevant. Pipes originated on the earliest Unix machine,
>>>>>> which was a PDP-11 with only a 64kiB address space. They work great
>>>>>> for pumping around gigabytes of data, but you don???t need
>>>>>> gigabyte-sized memory buffers to do that.
>>>>>
>>>>> It all has to be SOMEWHERE ... if not in RAM then
>>>>> on a mass storage device.
>>>>
>>>> Nope, at least not with pipes.
>>>
>>> Hold on a sec.... pipes are /buffered/ in RAM, so there's at least
>>> a small bit of ram set aside for each open pipe.
>>
>> The word ???all??? isn???t just decoration. The claim was ???it all has to be
>> somewhere???, and Rich???s point (as I understand it) is that it does not
>> all have to be somewhere.
>>
>> For example,
>>
>> head -c $((1024*1024*1024)) /dev/urandom | sha256sum
>>
>> puts a gigabyte of data through a pipe, but at no point does anything
>> allocate anywhere close to a gigabyte of storage of any kind.
>
>
> There's truth here ... the gigabyte gets kinda 'spread out'.
>
> But what if it was ten GB, or 100 ?
>
> So, if you wanna count on sending vast quantities of
> data in pipes, go ahead - it'll almost always work.
>
> Until vids go 16k or something ...
>
> As said LONG back ... each developer has to make
> decisions based on their app and the 'likely'
> systems/usage. There isn't really a 'wrong' or
> 'right' way here - just 'alternatives'. I like
> terse pipes, you may not care.

You keep ignoring the piped-to program, it's running concurrently
(or with multiple cores, simultaneously) as long as it keeps reading
from the pipe the pipe will never fill up.