Phillip Frabott <nntp@fulltermprivacy.com> writes:
> On 11/21/2024 13:38, Richard Kettlewell wrote:
>> This remark makes me wonder if you’ve got the wrong end of the stick
>> about what a named pipe is. They are really not the same as regular
>> files, temporary or otherwise.
>
> From a performance perspective they are. At least from the work I've
> done. We had to drop named pipes solely because of the performance hit
> because it is writing to a file system

The only filesystem write involve is the initial mkfifo(at) call. No
further filesystem writes are involved.

--
https://www.greenend.org.uk/rjk/

On 11/22/2024 03:38, vallor wrote:
> On 22 Nov 2024 07:29:16 GMT, vallor <vallor@cultnix.org> wrote in
> <lqaq6cF8btnU3@mid.individual.net>:
>
>> However, the speed appears to be limited by dd in my examples -- setting
>> a block size to fill the pipe/packets seems to increase throughput:
>>
>> $ nc -l -U -N /tmp/socket > /dev/null & time -p ( dd if=/dev/zero
>> count=$[1024*1024*4] bs=1024 | nc -U -N /tmp/socket > /dev/null )
>
> Realized the bottleneck would be the pipe between dd and nc, so wrote
> a program to connect to /tmp/socket and spew data at it -- it sends
> 46950 212992-byte buffers (9999974400 bytes) in 2.41 seconds.
> (4149366971 bytes/second, or 4.1GB/s).
>
> (The default "MTU" for a Linux Unix socket connection
> is 212992 bytes. Default pipe size is 8*512 bytes.)
>

Yeah, this has been the experience of our testing as well. We can also
bump the MTU up a bit to get sockets to do better and we've also
modified some of the socket code to our environment so it's able to do
around 12GB/s. We think we can tune it further to get about 15GB/s from
sockets. At the moment we have to do 4.6GB per 0.5sec to keep up, and
7GB per 0.5sec to stay ahead. We've done everything we can to squeeze
out as much performance from pipes as we could but there just isn't much
we can do anymore. But yeah, sockets are way more capable of large data
intake vs pipes, and this shows as well in a lot of research that's been
done. We've been testing sockets for about the last year and the results
have shown we are keeping up better then we are with pipes right now.

And for tribute to pipes, we've been using pipes in this project for the
last 30 years. That's a pretty great track record for something to have
been useful for such a long time and not be replaced with something else
until now. Sadly sockets will likely bottom out in the next 10-15 years
and we'll have to switch to something else. But for now, it's working.

--
Phillip Frabott
----------
- Adam: Is a void really a void if it returns?
- Jack: No, it's just nullspace at that point.
----------

vallor <vallor@cultnix.org> wrote:
> On Fri, 22 Nov 2024 03:12:43 -0000 (UTC), Lawrence D'Oliveiro
> <ldo@nz.invalid> wrote in <vhosra$1171f$1@dont-email.me>:
>
>> On Thu, 21 Nov 2024 21:55:37 -0500, Phillip Frabott wrote:
>>
>>> We had to drop named pipes solely because of the performance hit
>>> because it is writing to a file system so it's being controlled by the
>>> file system, even if that file system is in memory.
>>
>> That doesn’t make any sense, if we were talking about Linux. Is this on
>> Windows, by any chance?
>
> Doesn't the named pipe connection work through the filesystem code?
> That could add overhead.

Only to the extent that a filesystem lookup has to occur to lookup the
name in order to open() the name.

Once you have a file descriptor back from the open() call, there is no
difference at all kernel wise betwenn the two, they are one and the
same block of kernel code.

> Can't use named pipes on just any filesystem -- won't work on NFS
> for example, unless I'm mistaken.

Correct, you need a filesystem that supports storing a 'name' that
that is a reference to a pipe, so windows filesystems are out.

Named pipes appear as 'pipe' nodes across NFS (just tested this to be
certian). And, so long as all the "accessors" of the named pipe are
running on the same Linux machine with the NFS mount containing the
pipe node, the named pipe works as expected (just tested this as well).

But a named pipe on NFS does not give you a machine to machine (two
different machines) transmit channel.

>>> As the demand grows, we are actually at the limits of performance that
>>> even unnamed pipes gives us. So we are starting to migrate to UNIX
>>> sockets which has about double to bandwidth and performance of pipes.
>>
>> Not sure how that works, given that Unix sockets are actually a more
>> complex mechanism than pipes.
>
> With Unix sockets, once the connection is made, it's all in-memory
> networking.

Correct.

> I suspect (but don't know) that named pipes require the data to pass
> through the filesystem for each write.

Incorrect. The only 'filesystem' access for named pipes is during the
open() call to look up the name from the filesystem. Once you get the
file descriptor back, it is the exact same in-memory FIFO queue as an
anonymous pipe created via pipe() (at least on Linux).

On Fri, 22 Nov 2024 10:00:31 -0500, Phillip Frabott wrote:

> At the moment we have to do 4.6GB per 0.5sec to keep up, and
> 7GB per 0.5sec to stay ahead. We've done everything we can to squeeze
> out as much performance from pipes as we could but there just isn't much
> we can do anymore.

Sounds like a job for a shared memory section.

On 11/22/2024 15:52, Lawrence D'Oliveiro wrote:
> On Fri, 22 Nov 2024 10:00:31 -0500, Phillip Frabott wrote:
>
>> At the moment we have to do 4.6GB per 0.5sec to keep up, and
>> 7GB per 0.5sec to stay ahead. We've done everything we can to squeeze
>> out as much performance from pipes as we could but there just isn't much
>> we can do anymore.
>
> Sounds like a job for a shared memory section.

That only works if the entire dataset originates on, and results within,
the same system. Unfortunately this is not the case so it wouldn't be
possible. This is an intermediate system between the intake and final
processing systems which are further down the line.

Unless you can recommend a way to do shared memory across different
machines without extra hardware. Currently we ingest data from a
(external) /dev (unbuffered data stream) into a process on the machine
that buffers the data (currently though an unnamed pipe, which as I said
before is in the process of being replaced with UNIX socket) then that
data pipes to another process (again, being replaced with UNIX socket
soon) which converts the data stream from the buffer to block data and
then categorizes those blocks based on data types and depending on those
data types sends it to one of 4 /dev devices that leave the machine. The
bottleneck is not between the first and second process (buffer process
and stream to block data processor) it's between the intake /dev to the
intermediate and the intermediate to the 4 outbound /devs. These /dev
are not networks so there is no TCP data. They are hard lined between
the /dev machine and the intermediate computer and the intermediate
computer to the 4 other /dev machines. So how would I share memory
between /dev machines from the 80s-90s and the intermediate computer
without using any network or TCP/UDP capabilities?

(If it was not apparent. these are manufacturing machines.)

--
Phillip Frabott
----------
- Adam: Is a void really a void if it returns?
- Jack: No, it's just nullspace at that point.
----------

On Fri, 22 Nov 2024 17:06:06 -0500, Phillip Frabott wrote:

> Unless you can recommend a way to do shared memory across different
> machines without extra hardware.

Ah, I thought you were running all these processes on the same machine,
since pipes don’t work any other way.

What about those high-speed interconnects like the ones they use in
supercomputers? 40Gb Ethernet (not sure if 100Gb is working yet)? How fast
does SCSI go? Some fibre-based interconnect?

On Fri, 22 Nov 2024 19:11:23 -0000 (UTC), Rich <rich@example.invalid>
wrote in <vhql0r$1a0ch$2@dont-email.me>:

> vallor <vallor@cultnix.org> wrote:
>> On Fri, 22 Nov 2024 03:12:43 -0000 (UTC), Lawrence D'Oliveiro
>> <ldo@nz.invalid> wrote in <vhosra$1171f$1@dont-email.me>:
>>
>>> On Thu, 21 Nov 2024 21:55:37 -0500, Phillip Frabott wrote:
>>>
>>>> We had to drop named pipes solely because of the performance hit
>>>> because it is writing to a file system so it's being controlled by
>>>> the file system, even if that file system is in memory.
>>>
>>> That doesn’t make any sense, if we were talking about Linux. Is this
>>> on Windows, by any chance?
>>
>> Doesn't the named pipe connection work through the filesystem code?
>> That could add overhead.
>
> Only to the extent that a filesystem lookup has to occur to lookup the
> name in order to open() the name.
>
> Once you have a file descriptor back from the open() call, there is no
> difference at all kernel wise betwenn the two, they are one and the same
> block of kernel code.

I stand corrected about that.

>
>> Can't use named pipes on just any filesystem -- won't work on NFS for
>> example, unless I'm mistaken.
>
> Correct, you need a filesystem that supports storing a 'name' that that
> is a reference to a pipe, so windows filesystems are out.
>
> Named pipes appear as 'pipe' nodes across NFS (just tested this to be
> certian). And, so long as all the "accessors" of the named pipe are
> running on the same Linux machine with the NFS mount containing the pipe
> node, the named pipe works as expected (just tested this as well).

I tested it too (with an NFS v4.1 filesystem), and yes, mkfifo makes
a named pipe, and it works as expected. (Didn't expect it to work
across machines, though that would be a neat trick.)

> But a named pipe on NFS does not give you a machine to machine (two
> different machines) transmit channel.
>
>>>> As the demand grows, we are actually at the limits of performance
>>>> that even unnamed pipes gives us. So we are starting to migrate to
>>>> UNIX sockets which has about double to bandwidth and performance of
>>>> pipes.
>>>
>>> Not sure how that works, given that Unix sockets are actually a more
>>> complex mechanism than pipes.
>>
>> With Unix sockets, once the connection is made, it's all in-memory
>> networking.
>
> Correct.
>
>> I suspect (but don't know) that named pipes require the data to pass
>> through the filesystem for each write.
>
> Incorrect. The only 'filesystem' access for named pipes is during the
> open() call to look up the name from the filesystem. Once you get the
> file descriptor back, it is the exact same in-memory FIFO queue as an
> anonymous pipe created via pipe() (at least on Linux).

Again, I stand corrected on that.

(Haven't figured out how to increase ulimit -p yet, doesn't seem
to want to increase, even as root...)

--
-v System76 Thelio Mega v1.1 x86_64 NVIDIA RTX 3090 Ti
OS: Linux 6.12.0 Release: Mint 21.3 Mem: 258G
"A good hot dog feeds the hand that bites it."

vallor <vallor@cultnix.org> wrote:
> (Haven't figured out how to increase ulimit -p yet, doesn't seem
> to want to increase, even as root...)

From the ulimit manpage:

-p The pipe size in 512-byte blocks (this may not be set)

Note the "this may not be set" part at the end.

It is likely the case that if you wanted to increase the pipe size that
you would have to make a change in the kernel source and recompile a
custom kernel.

On 11/22/24 1:49 AM, Lawrence D'Oliveiro wrote:
> On Fri, 22 Nov 2024 01:44:35 -0500, 186282@ud0s4.net wrote:
>
>> On 11/21/24 4:56 PM, Lawrence D'Oliveiro wrote:
>>
>>> I have used them to transfer quite large amounts, quickly and reliably.
>>
>> Yep, WILL work. No question.
>>
>> The question is HOW MUCH should you intend to send back
>> and forth using pipes (or any other method) between the
>> parent and children.
>
> How about 10 gigabytes, which I was able to transfer in two seconds? Is
> that “too much” for you?

Yep - though you may have had a 'radical vision' when
writing your parent/child app .... just because *I*
wouldn't do it .......

256 Bytes is about as far as I'd go. The children/threads
should deal with the big files themselves IMHO. Put the
kids to work :-)

On 23 Nov 2024 03:25:08 GMT, vallor wrote:

> I tested it too (with an NFS v4.1 filesystem), and yes, mkfifo makes a
> named pipe, and it works as expected. (Didn't expect it to work across
> machines, though that would be a neat trick.)

No reason why it shouldn’t work, provided the network protocol has support
for recognizing such special files.

> (Haven't figured out how to increase ulimit -p yet, doesn't seem to want
> to increase, even as root...)

<https://manpages.debian.org/7/pipe.7.en.html> mentions the
/proc/sys/fs/pipe-max-size configuration limit. It could be that ulimit -p
has no effect on Linux.

On Fri, 22 Nov 2024 23:44:24 -0500, 186282@ud0s4.net wrote:

> On 11/22/24 1:49 AM, Lawrence D'Oliveiro wrote:
>> On Fri, 22 Nov 2024 01:44:35 -0500, 186282@ud0s4.net wrote:
>>
>>> On 11/21/24 4:56 PM, Lawrence D'Oliveiro wrote:
>>>
>>>> I have used them to transfer quite large amounts, quickly and
>>>> reliably.
>>>
>>> Yep, WILL work. No question.
>>>
>>> The question is HOW MUCH should you intend to send back and forth
>>> using pipes (or any other method) between the parent and children.
>>
>> How about 10 gigabytes, which I was able to transfer in two seconds? Is
>> that “too much” for you?
>
> Yep - though you may have had a 'radical vision' when writing your
> parent/child app .... just because *I* wouldn't do it .......

You didn’t realize IPC on *nix is industrial-strength?

On 11/23/24 12:19 AM, Lawrence D'Oliveiro wrote:
> On Fri, 22 Nov 2024 23:44:24 -0500, 186282@ud0s4.net wrote:
>
>> On 11/22/24 1:49 AM, Lawrence D'Oliveiro wrote:
>>> On Fri, 22 Nov 2024 01:44:35 -0500, 186282@ud0s4.net wrote:
>>>
>>>> On 11/21/24 4:56 PM, Lawrence D'Oliveiro wrote:
>>>>
>>>>> I have used them to transfer quite large amounts, quickly and
>>>>> reliably.
>>>>
>>>> Yep, WILL work. No question.
>>>>
>>>> The question is HOW MUCH should you intend to send back and forth
>>>> using pipes (or any other method) between the parent and children.
>>>
>>> How about 10 gigabytes, which I was able to transfer in two seconds? Is
>>> that “too much” for you?
>>
>> Yep - though you may have had a 'radical vision' when writing your
>> parent/child app .... just because *I* wouldn't do it .......
>
> You didn’t realize IPC on *nix is industrial-strength?

I know it CAN handle BIG transactions.

But SHOULD it ?

This is maybe a "philosophical" matter ...

I'm still of the crap CPU/Mem era ... always look
to minimize/simplify.

On Sat, 23 Nov 2024 01:27:25 -0500, 186282@ud0s4.net wrote:

> On 11/23/24 12:19 AM, Lawrence D'Oliveiro wrote:
>>
>> You didn’t realize IPC on *nix is industrial-strength?
>
> I know it CAN handle BIG transactions.
>
> But SHOULD it ?

That’s what it’s designed for!

On 23/11/2024 06:27, 186282@ud0s4.net wrote:

>
>   I know it CAN handle BIG transactions.
>
>   But SHOULD it ?
>
>   This is maybe a "philosophical" matter ...
>

*Should* is an interesting word, used today in enormous quantities, to
justify actions.
It is a nasty little fella, because its roots like in the language of
debt - both physical and moral - and obligation.

And whilst it seems to not overtly contain such elements, implicitly it
does.

Do if I say 'you *should* do this, there is an implicit quality of
(unspecified) goodness and badness and obligation on you in doing it or
not doing it.

Unless you prefix it with a conditional like 'If you want to get to
Rome, you should take this road' there is the implicit conditional of
'if you want to be regarded as a good and holy person' you should take
this road.

>   I'm still of the crap CPU/Mem era ... always look
>   to minimize/simplify.

Well yes, but we have gigabytes of RAM these days.

You have to look at what your are doing, what you are trying to do, and
what resources are available, to do it with, how long it will take and
what it will cost.

This is standard professional engineering philosophy. It is also officer
class military procedure - a pragmatic approach to achieving well
defined tactical and strategic objectives.

You cant ask 'should we' in isolation.
Only in the context of the above.

--
The theory of Communism may be summed up in one sentence: Abolish all
private property.

Karl Marx

On 2024-11-23 08:41, Lawrence D'Oliveiro wrote:
> On Sat, 23 Nov 2024 01:27:25 -0500, 186282@ud0s4.net wrote:
>
>> On 11/23/24 12:19 AM, Lawrence D'Oliveiro wrote:
>>>
>>> You didn’t realize IPC on *nix is industrial-strength?
>>
>> I know it CAN handle BIG transactions.
>>
>> But SHOULD it ?
>
> That’s what it’s designed for!

I use named pipes on my backup script.

I dd a hard disk partition, compress it, and at the same time calculate a checksum.

mkfifo mdpipe
dd if=/dev/$1 status=progress bs=16M | tee mdpipe | pigz -3 > $3.gz &
md5sum -b mdpipe | tee -a md5checksum_expanded
wait
rm mdpipe
echo "$3" >> md5checksum_expanded

This way there is only one disk read operation. I can see the thing running at max hard disk speed.

--
Cheers, Carlos.

On Sat, 23 Nov 2024 11:12:50 +0000, The Natural Philosopher wrote:

> *Should* is an interesting word ...

Here I think it is being used to backpedal from the poster’s original
claim that pipes are somehow unsuited to passing around large quantities
of data, while trying to somehow save face.

> Well yes, but we have gigabytes of RAM these days.

That’s irrelevant. Pipes originated on the earliest Unix machine, which
was a PDP-11 with only a 64kiB address space. They work great for pumping
around gigabytes of data, but you don’t need gigabyte-sized memory buffers
to do that.

Carlos E.R. <robin_lis...@es.invalid> [CE]:
CE> I dd a hard disk partition, compress it, and at the same time
CE> calculate a checksum.
CE>
CE> mkfifo mdpipe
CE> dd if=/dev/$1 status=progress bs=16M | tee mdpipe | pigz -3 > $3.gz &
CE> md5sum -b mdpipe | tee -a md5checksum_expanded
CE> wait
CE> rm mdpipe
CE> echo "$3" >> md5checksum_expanded

David B Rosen has written the tpipe(1) utility for exactly such cases:

The above steps can be rewritten in a much cleaner way as:

dd if=/dev/$1 status=progress bs=16M |
tpipe "md5sum -b >> md5checksum_expanded" |
pigz -3 > $3.gz

If tpipe is not available on your system, you can always use the shell's
process substitution feature instead:

dd if=/dev/$1 status=progress bs=16M |
tee >(md5sum -b >> md5checksum_expanded) \
>(pigz -3 > $3.gz) \
>/dev/null

On Sat, 23 Nov 2024 14:39:40 +0100, Carlos E.R. wrote:

> I use named pipes on my backup script.
>
> I dd a hard disk partition, compress it, and at the same time calculate
> a checksum.
>
> mkfifo mdpipe
> dd if=/dev/$1 status=progress bs=16M | tee mdpipe | pigz -3 > $3.gz &
> md5sum -b mdpipe | tee -a md5checksum_expanded
> wait
> rm mdpipe
> echo "$3" >> md5checksum_expanded
>
> This way there is only one disk read operation. I can see the thing
> running at max hard disk speed.

Clever. Just one thing, I would probably use a dynamic name for the
pipe and put it in $TMPDIR (e.g. generated with tempfile) so that 1)
multiple instances could run at once, and 2) it doesn’t depend on
writing into the current directory, whatever that might be.

(It’s likely neither of those issues is relevant to your particular
use case ...)

On 2024-11-23 23:04, Lawrence D'Oliveiro wrote:
> On Sat, 23 Nov 2024 14:39:40 +0100, Carlos E.R. wrote:
>
>> I use named pipes on my backup script.
>>
>> I dd a hard disk partition, compress it, and at the same time calculate
>> a checksum.
>>
>> mkfifo mdpipe
>> dd if=/dev/$1 status=progress bs=16M | tee mdpipe | pigz -3 > $3.gz &
>> md5sum -b mdpipe | tee -a md5checksum_expanded
>> wait
>> rm mdpipe
>> echo "$3" >> md5checksum_expanded
>>
>> This way there is only one disk read operation. I can see the thing
>> running at max hard disk speed.
>
> Clever. Just one thing, I would probably use a dynamic name for the
> pipe and put it in $TMPDIR (e.g. generated with tempfile) so that 1)
> multiple instances could run at once, and 2) it doesn’t depend on
> writing into the current directory, whatever that might be.
>
> (It’s likely neither of those issues is relevant to your particular
> use case ...)

No, it is not relevant :-)

Using the current dir allows me to see it there, too.

--
Cheers, Carlos.

On 2024-11-23 22:32, Fritz Wuehler wrote:
> Carlos E.R. <robin_lis...@es.invalid> [CE]:
> CE> I dd a hard disk partition, compress it, and at the same time
> CE> calculate a checksum.
> CE>
> CE> mkfifo mdpipe
> CE> dd if=/dev/$1 status=progress bs=16M | tee mdpipe | pigz -3 > $3.gz &
> CE> md5sum -b mdpipe | tee -a md5checksum_expanded
> CE> wait
> CE> rm mdpipe
> CE> echo "$3" >> md5checksum_expanded
>
>
> David B Rosen has written the tpipe(1) utility for exactly such cases:
>
> The above steps can be rewritten in a much cleaner way as:
>
> dd if=/dev/$1 status=progress bs=16M |
> tpipe "md5sum -b >> md5checksum_expanded" |
> pigz -3 > $3.gz
>
>
> If tpipe is not available on your system,

it is not.

Telcontar:~ # tpipe
If 'tpipe' is not a typo you can use command-not-found to lookup the
package that contains it, like this:
cnf tpipe
Telcontar:~ # cnf tpipe
tpipe: command not found
Telcontar:~ # opi tpipe
Searching repos for: tpipe
1. inputpipe
2. socketpipe
3. inputpipe-debuginfo
4. socketpipe-debuginfo
5. inputpipe-debugsource
6. socketpipe-debugsource
Pick a number (0 to quit): 0
Telcontar:~ #

> you can always use the shell's
> process substitution feature instead:
>
> dd if=/dev/$1 status=progress bs=16M |
> tee >(md5sum -b >> md5checksum_expanded) \
> >(pigz -3 > $3.gz) \
> >/dev/null
>

I like my way, it is mine :-)

--
Cheers, Carlos.

On 11/23/24 4:25 PM, Lawrence D'Oliveiro wrote:
> On Sat, 23 Nov 2024 11:12:50 +0000, The Natural Philosopher wrote:
>
>> *Should* is an interesting word ...
>
> Here I think it is being used to backpedal from the poster’s original
> claim that pipes are somehow unsuited to passing around large quantities
> of data, while trying to somehow save face.

Check - I said there were OTHER, reasonably good, ways
to pass lots of data between parents/children. Others
here said they've used ordinary disk files instead of
pipes for various reasons.

As for my face ... it never was very handsome, no
saving it now ... :-)

>> Well yes, but we have gigabytes of RAM these days.
>
> That’s irrelevant. Pipes originated on the earliest Unix machine, which
> was a PDP-11 with only a 64kiB address space. They work great for pumping
> around gigabytes of data, but you don’t need gigabyte-sized memory buffers
> to do that.

It all has to be SOMEWHERE ... if not in RAM then
on a mass storage device.

Of course 'gigabytes' ALL AT ONCE -vs- "a little
at a time, added up" are entirely different things.

On Sat, 23 Nov 2024 19:42:01 -0500, 186282@ud0s4.net wrote:

> On 11/23/24 4:25 PM, Lawrence D'Oliveiro wrote:
>
>> Pipes originated on the earliest Unix machine, which
>> was a PDP-11 with only a 64kiB address space. They work great for
>> pumping around gigabytes of data, but you don’t need gigabyte-sized
>> memory buffers to do that.
>
> It all has to be SOMEWHERE ... if not in RAM then on a mass storage
> device.

It might be generated in one process and consumed in another. It might be
coming from the network, or going to the network -- the process at the
other end of the pipe being isolated from the network, perhaps for
security reasons.

In short, it might work any number of ways, without involving (local)
mass-storage devices.

186282@ud0s4.net <186283@ud0s4.net> wrote:
> On 11/23/24 4:25 PM, Lawrence D'Oliveiro wrote:
>> That’s irrelevant. Pipes originated on the earliest Unix machine,
>> which was a PDP-11 with only a 64kiB address space. They work great
>> for pumping around gigabytes of data, but you don’t need
>> gigabyte-sized memory buffers to do that.
>
> It all has to be SOMEWHERE ... if not in RAM then
> on a mass storage device.

Nope, at least not with pipes.

> Of course 'gigabytes' ALL AT ONCE -vs- "a little at a time, added
> up" are entirely different things.

"A little at a time, added up" is exactly what pipes give you, and is
what allows processing gigabytes of total data, without also needing
gigabytes of memory (or disk) to do the processing.

I regularly re-encode video files using a pipeline of several image
filters from the mjpegtools package, ending up with a pipe to x264 for
final compression (for size reduction purposes).

The source files tend to be around 2-5G. The pipes between the image
filters, and the final pipe to the x264 compressor transfers raw
uncompressed YUV data.

The raw uncompressed YUV data for an hours worth of 1080p video is
about 336GiB of data (way more data than I have RAM to store it all,
and slightly more than the total free space available right now on the
1.9T disk I run the encodes from). Not one byte of that 336GiB of raw
YUV data that passes through the pipes ever touches disk storage. The
only data in disk files are the source, compressed, video file, and the
final result, also compressed, video file.

On Sun, 24 Nov 2024 14:25:10 +0000, Rich wrote:

> 186282@ud0s4.net <186283@ud0s4.net> wrote:
>> On 11/23/24 4:25 PM, Lawrence D'Oliveiro wrote:
>>> That’s irrelevant. Pipes originated on the earliest Unix machine,
>>> which was a PDP-11 with only a 64kiB address space. They work great
>>> for pumping around gigabytes of data, but you don’t need
>>> gigabyte-sized memory buffers to do that.
>>
>> It all has to be SOMEWHERE ... if not in RAM then
>> on a mass storage device.
>
> Nope, at least not with pipes.

Hold on a sec.... pipes are /buffered/ in RAM, so there's at least
a small bit of ram set aside for each open pipe. On Linux, pipe(7)
says "In Linux versions before 2.6.11, the capacity of a pipe was
the same as the system page size (e.g., 4096 bytes on i386).
Since Linux 2.6.11, the pipe capacity is 65536 bytes. Since
Linux 2.6.35, the default pipe capacity is 65536 bytes, but
the capacity can be queried and set using the fcntl(2)
F_GETPIPE_SZ and F_SETPIPE_SZ operations. See fcntl(2) for
more information."
and that "capacity" referred to consists of a kernel-managed RAM buffer.

[snip]

--
Lew Pitcher
"In Skills We Trust"

Lew Pitcher <lew.pitcher@digitalfreehold.ca> wrote:
> On Sun, 24 Nov 2024 14:25:10 +0000, Rich wrote:
>
>> 186282@ud0s4.net <186283@ud0s4.net> wrote:
>>> On 11/23/24 4:25 PM, Lawrence D'Oliveiro wrote:
>>>> That’s irrelevant. Pipes originated on the earliest Unix machine,
>>>> which was a PDP-11 with only a 64kiB address space. They work
>>>> great for pumping around gigabytes of data, but you don’t need
>>>> gigabyte-sized memory buffers to do that.
>>>
>>> It all has to be SOMEWHERE ... if not in RAM then
>>> on a mass storage device.
>>
>> Nope, at least not with pipes.
>
> Hold on a sec.... pipes are /buffered/ in RAM, so there's at least a
> small bit of ram set aside for each open pipe. On Linux, pipe(7)
> says "In Linux versions before 2.6.11, the capacity of a pipe was the
> same as the system page size (e.g., 4096 bytes on i386). Since Linux
> 2.6.11, the pipe capacity is 65536 bytes. Since Linux 2.6.35, the
> default pipe capacity is 65536 bytes, but the capacity can be queried
> and set using the fcntl(2) F_GETPIPE_SZ and F_SETPIPE_SZ operations.
> See fcntl(2) for more information." and that "capacity" referred to
> consists of a kernel-managed RAM buffer.
>
> [snip]

Fair enough -- about 64k of RAM per each pipe is used for the kernel
FIFO buffer.

But that is at most 64k of RAM (unless one's C code adjusts the buffer
size), regardless of how many GiB flow through the pipe. So it "all"
(to quote the nymshifter) does not have to be in RAM.