There has been a long tendency in Linux/comp groups
to horribly attack, belittle, abuse, anybody who has
had a 'different experience' and sees things a little
off some kinda poorly-defined 'norm'.

Sorry, we all didn't come up on the same track.
A thousand different paths, a thousand different
styles of apps/needs/solutions. Computers let
you DO that.

Hey, if I feel the need to use files instead
of 'pipes' then I WANT TO USE FILES INSTEAD
OF PIPES. Don't like it ? Tuff titty. Any
'contributions' will be to tell me how to
maximize that approach - not to rain down piss.

(actually I think pipes are better - but there
are and will be other approaches/reasons)

The recent elections kinda upset the Linux/Unix
groups - a lot of politics promoted a lot of
threads. Well, the elections are OVER now.
Back to business.

BUT ... consider ... "back to business" does
not need to mean "back to old habits". We all
can do BETTER, move towards the future instead
of being at each others throats over NOTHING.

Just sayin'

Most everybody here seems to have been in the
groove since (or during) PUNCH CARDS. Let's
not be petty. We all did it OUR WAY.

Hey, I remember the giant handful of punch
cards - DON'T DROP 'EM ! :-)

--
033-33

On Mon, 18 Nov 2024, 186282@ud0s4.net wrote:

> There has been a long tendency in Linux/comp groups
> to horribly attack, belittle, abuse, anybody who has
> had a 'different experience' and sees things a little
> off some kinda poorly-defined 'norm'.
>
> Sorry, we all didn't come up on the same track.
> A thousand different paths, a thousand different
> styles of apps/needs/solutions. Computers let
> you DO that.
>
> Hey, if I feel the need to use files instead
> of 'pipes' then I WANT TO USE FILES INSTEAD
> OF PIPES. Don't like it ? Tuff titty. Any
> 'contributions' will be to tell me how to
> maximize that approach - not to rain down piss.
>
> (actually I think pipes are better - but there
> are and will be other approaches/reasons)
>
> The recent elections kinda upset the Linux/Unix
> groups - a lot of politics promoted a lot of
> threads. Well, the elections are OVER now.
> Back to business.
>
> BUT ... consider ... "back to business" does
> not need to mean "back to old habits". We all
> can do BETTER, move towards the future instead
> of being at each others throats over NOTHING.
>
> Just sayin'
>
> Most everybody here seems to have been in the
> groove since (or during) PUNCH CARDS. Let's
> not be petty. We all did it OUR WAY.
>
> Hey, I remember the giant handful of punch
> cards - DON'T DROP 'EM ! :-)

Are you saying we should disregard the emperor? Doesn't he teach us that
our hate makes us stronger?

On 11/18/24 06:20, 186282@ud0s4.net wrote:

> Most everybody here seems to have been in the
> groove since (or during) PUNCH CARDS. Let's
> not be petty. We all did it OUR WAY.
>
> Hey, I remember the giant handful of punch
> cards - DON'T DROP 'EM !  :-)
>

When I started Uni they showed us their computers with punch cards, so I
avoided the computers until they had terminals, vt100 at least.

Anyway make Linux great again.

"186282@ud0s4.net" <186283@ud0s4.net> writes:
> There has been a long tendency in Linux/comp groups
> to horribly attack, belittle, abuse, anybody who has
> had a 'different experience' and sees things a little
> off some kinda poorly-defined 'norm'.
>
> Sorry, we all didn't come up on the same track.
> A thousand different paths, a thousand different
> styles of apps/needs/solutions. Computers let
> you DO that.
>
> Hey, if I feel the need to use files instead
> of 'pipes' then I WANT TO USE FILES INSTEAD
> OF PIPES. Don't like it ? Tuff titty. Any
> 'contributions' will be to tell me how to
> maximize that approach - not to rain down piss.

Precisely nobody is telling you that you can’t use files.

If the issue here is that you don’t like being told that pipes aren’t
the same as regular files, well, maybe consider not posting that they
are the same?

--
https://www.greenend.org.uk/rjk/

186282@ud0s4.net <186283@ud0s4.net> wrote:
> Hey, if I feel the need to use files instead of 'pipes' then I WANT
> TO USE FILES INSTEAD OF PIPES.

Except you did not say you were "using files instead of 'pipes'".

What you said, in Message-ID: <hzSdnTUBKbG_YKv6nZ2dnZfqnPQAAAAA@earthlink.com>
was:

186282@ud0s4.net <186283@ud0s4.net> wrote:
>...
>
> Pipes are good.
>
> But, really, they're just temp files the parent process can access.

When reality is that pipes have not been "just temp files" since the
day's of MSDOS's fake "pipes", and for Unix systems, pipes have never
been "just temp files".

And when challenged, you doubled down on the just temp files parts.

On 11/18/2024 05:46, Rich wrote:
> 186282@ud0s4.net <186283@ud0s4.net> wrote:
>> Hey, if I feel the need to use files instead of 'pipes' then I WANT
>> TO USE FILES INSTEAD OF PIPES.
>
> Except you did not say you were "using files instead of 'pipes'".
>
> What you said, in Message-ID: <hzSdnTUBKbG_YKv6nZ2dnZfqnPQAAAAA@earthlink.com>
> was:
>
> 186282@ud0s4.net <186283@ud0s4.net> wrote:
>> ...
>>
>> Pipes are good.
>>
>> But, really, they're just temp files the parent process can access.
>
> When reality is that pipes have not been "just temp files" since the
> day's of MSDOS's fake "pipes", and for Unix systems, pipes have never
> been "just temp files".
>
> And when challenged, you doubled down on the just temp files parts.

I think the point that is being made by calling pipes a "temp files" is
that they are not persistent. When the computer is turned off they
disappear (unlike files on hard (persistent) storage). Granted a pipe
might re-appear later when the system is back up and running but my
professor always said, don't expect any guarantees with pipes. They may
be there, they may not be there, you might be able to use them one
minute and they disappear the next. Consider them temporary aberrations
until proven otherwise.

Chances are that the pipes of today are more solid then the pipes of
yesteryear, but we've all grown up at different times with different
states of computing and different upbringings. There are pros and cons
to everything, including using files AND using pipes. To each their own.

I wouldn't argue about the small things like this.

--
Phillip Frabott
----------
- Adam: Is a void really a void if it returns?
- Jack: No, it's just nullspace at that point.
----------

On Mon, 18 Nov 2024 01:20:53 -0500, 186282@ud0s4.net wrote:

> Hey, if I feel the need to use files instead of 'pipes' then I WANT TO
> USE FILES INSTEAD OF PIPES.

Except ... you were trying to argue that there was no fundamental
difference between pipes and files anyway. That you could somehow do
everything you could do with pipes by using temporary files.

On Mon, 18 Nov 2024 09:45:07 -0500, Phillip Frabott wrote:

> I think the point that is being made by calling pipes a "temp files" is
> that they are not persistent.

Named pipes can indeed be persistent.

On 11/18/24 4:13 AM, D wrote:
>
>
> On Mon, 18 Nov 2024, 186282@ud0s4.net wrote:
>
>> There has been a long tendency in Linux/comp groups
>> to horribly attack, belittle, abuse, anybody who has
>> had a 'different experience' and sees things a little
>> off some kinda poorly-defined 'norm'.
>>
>> Sorry, we all didn't come up on the same track.
>> A thousand different paths, a thousand different
>> styles of apps/needs/solutions. Computers let
>> you DO that.
>>
>> Hey, if I feel the need to use files instead
>> of 'pipes' then I WANT TO USE FILES INSTEAD
>> OF PIPES. Don't like it ? Tuff titty. Any
>> 'contributions' will be to tell me how to
>> maximize that approach - not to rain down piss.
>>
>> (actually I think pipes are better - but there
>> are and will be other approaches/reasons)
>>
>> The recent elections kinda upset the Linux/Unix
>> groups - a lot of politics promoted a lot of
>> threads. Well, the elections are OVER now.
>> Back to business.
>>
>> BUT ... consider ... "back to business" does
>> not need to mean "back to old habits". We all
>> can do BETTER, move towards the future instead
>> of being at each others throats over NOTHING.
>>
>> Just sayin'
>>
>> Most everybody here seems to have been in the
>> groove since (or during) PUNCH CARDS. Let's
>> not be petty. We all did it OUR WAY.
>>
>> Hey, I remember the giant handful of punch
>> cards - DON'T DROP 'EM !  :-)
>
> Are you saying we should disregard the emperor? Doesn't he teach us that
> our hate makes us stronger?

Well, the 'woke' really did try to show us the
power of higher, gigabuck-funded, hate :-)

But that's all over for now.

In any case, I prefer to see these comp groups
as being much better when they are collaborative,
rather than derogative. Be you old boy or newbie,
everybody has a different 'vision', a slightly
different take on 'how it should be done'. Adding
1000 cuts and ad-homs - all too common on usenet -
does not represent any kind of improvement.

Just wanted to say it.

On Tue, 19 Nov 2024, 186282@ud0s4.net wrote:

> On 11/18/24 4:13 AM, D wrote:
>>
>>
>> On Mon, 18 Nov 2024, 186282@ud0s4.net wrote:
>>
>>> There has been a long tendency in Linux/comp groups
>>> to horribly attack, belittle, abuse, anybody who has
>>> had a 'different experience' and sees things a little
>>> off some kinda poorly-defined 'norm'.
>>>
>>> Sorry, we all didn't come up on the same track.
>>> A thousand different paths, a thousand different
>>> styles of apps/needs/solutions. Computers let
>>> you DO that.
>>>
>>> Hey, if I feel the need to use files instead
>>> of 'pipes' then I WANT TO USE FILES INSTEAD
>>> OF PIPES. Don't like it ? Tuff titty. Any
>>> 'contributions' will be to tell me how to
>>> maximize that approach - not to rain down piss.
>>>
>>> (actually I think pipes are better - but there
>>> are and will be other approaches/reasons)
>>>
>>> The recent elections kinda upset the Linux/Unix
>>> groups - a lot of politics promoted a lot of
>>> threads. Well, the elections are OVER now.
>>> Back to business.
>>>
>>> BUT ... consider ... "back to business" does
>>> not need to mean "back to old habits". We all
>>> can do BETTER, move towards the future instead
>>> of being at each others throats over NOTHING.
>>>
>>> Just sayin'
>>>
>>> Most everybody here seems to have been in the
>>> groove since (or during) PUNCH CARDS. Let's
>>> not be petty. We all did it OUR WAY.
>>>
>>> Hey, I remember the giant handful of punch
>>> cards - DON'T DROP 'EM !  :-)
>>
>> Are you saying we should disregard the emperor? Doesn't he teach us that
>> our hate makes us stronger?
>
>
> Well, the 'woke' really did try to show us the
> power of higher, gigabuck-funded, hate :-)
>
> But that's all over for now.
>
> In any case, I prefer to see these comp groups
> as being much better when they are collaborative,
> rather than derogative. Be you old boy or newbie,
> everybody has a different 'vision', a slightly
> different take on 'how it should be done'. Adding
> 1000 cuts and ad-homs - all too common on usenet -
> does not represent any kind of improvement.
>
> Just wanted to say it.
>
>

You are a wise man, 186282!

On 11/19/2024 19:23, Lawrence D'Oliveiro wrote:
> On Mon, 18 Nov 2024 09:45:07 -0500, Phillip Frabott wrote:
>
>> I think the point that is being made by calling pipes a "temp files" is
>> that they are not persistent.
>
> Named pipes can indeed be persistent.

Sure, but then your just creating a file with all the limitations that
come from that. IPC only benefits you when you use unnamed or
traditional pipes (performance and resources). Also, the pipe still
requires something on the other side of the transaction which might not
exist at some later point. Whereas a file on hard storage will almost
always be able to be transacted (assuming permissions of course)
regardless of what processes are (or are not) available. Also pipes also
require realtime processing, whereas storing things in a file can
provide for delayed/deferred processing at a later time.

Now, sure piping a command may be less typing, but there is no real
tangible benefit from the technical side from using a pipe vs typing the
command out over multiple lines of commands, the transaction remains the
same in both cases. But having non-persistent pipes gives you speed and
performance perks (and in most cases uses less RAM). So if your going to
do named pipes just know you aren't getting the real benefits of the IPC
if it's just going to write a file anyways.

--
Phillip Frabott
----------
- Adam: Is a void really a void if it returns?
- Jack: No, it's just nullspace at that point.
----------

On Thu, 21 Nov 2024 02:05:46 -0500, Phillip Frabott wrote:

> On 11/19/2024 19:23, Lawrence D'Oliveiro wrote:
>
>> On Mon, 18 Nov 2024 09:45:07 -0500, Phillip Frabott wrote:
>>
>>> I think the point that is being made by calling pipes a "temp files"
>>> is that they are not persistent.
>>
>> Named pipes can indeed be persistent.
>
> Sure, but then your just creating a file with all the limitations that
> come from that.

Not at all. It still has the same synchronization behaviour.

> IPC only benefits you when you use unnamed or traditional pipes
> (performance and resources).

Certainly not.

On 11/21/2024 02:22, Lawrence D'Oliveiro wrote:
> On Thu, 21 Nov 2024 02:05:46 -0500, Phillip Frabott wrote:
>
>> On 11/19/2024 19:23, Lawrence D'Oliveiro wrote:
>>
>>> On Mon, 18 Nov 2024 09:45:07 -0500, Phillip Frabott wrote:
>>>
>>>> I think the point that is being made by calling pipes a "temp files"
>>>> is that they are not persistent.
>>>
>>> Named pipes can indeed be persistent.
>>
>> Sure, but then your just creating a file with all the limitations that
>> come from that.
>
> Not at all. It still has the same synchronization behaviour.
>
>> IPC only benefits you when you use unnamed or traditional pipes
>> (performance and resources).
>
> Certainly not.

I guess it just depends on what you are doing. And in perspective, most
pipes are generally used for small amounts of data, the smaller then
data the less benefits you see between unnamed vs named pipes. I mean
100-bytes has zero performance differences between named and unnamed
while a 10MB pipe will always show that unnamed pipes are faster then
named pipes. So it's just depends on what you are doing and the data you
have. But as far as I know named pipes still go away when you turn the
machine off unless you are redirecting /tmp to hard storage.

And to be fair, I don't use named pipes as much as I used to so perhaps
things have changed. I've never had a good reason to use named pipes in
code in a very long time.

--
Phillip Frabott
----------
- Adam: Is a void really a void if it returns?
- Jack: No, it's just nullspace at that point.
----------

Phillip Frabott <nntp@fulltermprivacy.com> writes:
> On 11/21/2024 02:22, Lawrence D'Oliveiro wrote:
>> On Thu, 21 Nov 2024 02:05:46 -0500, Phillip Frabott wrote:
>>> On 11/19/2024 19:23, Lawrence D'Oliveiro wrote:
>>>> On Mon, 18 Nov 2024 09:45:07 -0500, Phillip Frabott wrote:
>>>>> I think the point that is being made by calling pipes a "temp files"
>>>>> is that they are not persistent.
>>>>
>>>> Named pipes can indeed be persistent.
>>>
>>> Sure, but then your just creating a file with all the limitations that
>>> come from that.

This remark makes me wonder if you’ve got the wrong end of the stick
about what a named pipe is. They are really not the same as regular
files, temporary or otherwise.

>> Not at all. It still has the same synchronization behaviour.
>>
>>> IPC only benefits you when you use unnamed or traditional pipes
>>> (performance and resources).
>> Certainly not.
>
> I guess it just depends on what you are doing. And in perspective,
> most pipes are generally used for small amounts of data, the smaller
> then data the less benefits you see between unnamed vs named pipes. I
> mean 100-bytes has zero performance differences between named and
> unnamed while a 10MB pipe will always show that unnamed pipes are
> faster then named pipes.

Apart from the details of how you get file descriptors to them, named
and anonymous pipes are identical.

Empirically (and unsurprisingly) there’s no performance difference in
Linux.

> So it's just depends on what you are doing and the data you have. But
> as far as I know named pipes still go away when you turn the machine
> off unless you are redirecting /tmp to hard storage.

The _contents_ of any kind of pipe go away when the last file descriptor
to them is closed (including rebooting or turning of the machine).

The _name_ of a named pipe goes away when it is unlinked or the
filesystem containing it is destroyed (which would include turning off
the machine, if the name is in a tmpfs).

--
https://www.greenend.org.uk/rjk/

On Thu, 21 Nov 2024 10:12:18 -0500, Phillip Frabott wrote:

> On 11/21/2024 02:22, Lawrence D'Oliveiro wrote:
>> On Thu, 21 Nov 2024 02:05:46 -0500, Phillip Frabott wrote:
>>
>>> On 11/19/2024 19:23, Lawrence D'Oliveiro wrote:
>>>
>>>> On Mon, 18 Nov 2024 09:45:07 -0500, Phillip Frabott wrote:
>>>>
>>>>> I think the point that is being made by calling pipes a "temp files"
>>>>> is that they are not persistent.
>>>>
>>>> Named pipes can indeed be persistent.
>>>
>>> Sure, but then your just creating a file with all the limitations that
>>> come from that.
>>
>> Not at all. It still has the same synchronization behaviour.
>>
>>> IPC only benefits you when you use unnamed or traditional pipes
>>> (performance and resources).
>>
>> Certainly not.
>
> I guess it just depends on what you are doing.

No it doesn’t. Named or not, pipes are pipes.

> And in perspective, most pipes are generally used for small amounts of
> data ...

I have used them to transfer quite large amounts, quickly and reliably.

On 11/21/2024 13:38, Richard Kettlewell wrote:
>
> This remark makes me wonder if you’ve got the wrong end of the stick
> about what a named pipe is. They are really not the same as regular
> files, temporary or otherwise.
>

From a performance perspective they are. At least from the work I've
done. We had to drop named pipes solely because of the performance hit
because it is writing to a file system so it's being controlled by the
file system, even if that file system is in memory. Anonymous (unnamed)
pipes seem to be different. Maybe it's the differences between how FIFO
works with named pipes over the regular pipes. I don't know, but we've
never been able to get named pipes to be 1:1 performance with unnamed pipes.

My best guess is that since unnamed pipes are not written to a file
system, it has a small performance advantage over named pipes. I deal
with large data streams that have very very small TTL's on them (in come
cases only a couple ms) so if a pipe takes too long it will get bogged
down and won't keep up. So it might be specific to the uses we have at
work. In probably 90% of everyone else's uses it's probably not noticeable.

As the demand grows, we are actually at the limits of performance that
even unnamed pipes gives us. So we are starting to migrate to UNIX
sockets which has about double to bandwidth and performance of pipes.

--
Phillip Frabott
----------
- Adam: Is a void really a void if it returns?
- Jack: No, it's just nullspace at that point.
----------

On Thu, 21 Nov 2024 21:55:37 -0500, Phillip Frabott wrote:

> We had to drop named pipes solely because of the performance hit
> because it is writing to a file system so it's being controlled by the
> file system, even if that file system is in memory.

That doesn’t make any sense, if we were talking about Linux. Is this on
Windows, by any chance?

> As the demand grows, we are actually at the limits of performance that
> even unnamed pipes gives us. So we are starting to migrate to UNIX
> sockets which has about double to bandwidth and performance of pipes.

Not sure how that works, given that Unix sockets are actually a more
complex mechanism than pipes.

On Fri, 22 Nov 2024 03:12:43 -0000 (UTC), Lawrence D'Oliveiro
<ldo@nz.invalid> wrote in <vhosra$1171f$1@dont-email.me>:

> On Thu, 21 Nov 2024 21:55:37 -0500, Phillip Frabott wrote:
>
>> We had to drop named pipes solely because of the performance hit
>> because it is writing to a file system so it's being controlled by the
>> file system, even if that file system is in memory.
>
> That doesn’t make any sense, if we were talking about Linux. Is this on
> Windows, by any chance?

Doesn't the named pipe connection work through the filesystem code?
That could add overhead.

Can't use named pipes on just any filesystem -- won't work on NFS
for example, unless I'm mistaken.

>
>> As the demand grows, we are actually at the limits of performance that
>> even unnamed pipes gives us. So we are starting to migrate to UNIX
>> sockets which has about double to bandwidth and performance of pipes.
>
> Not sure how that works, given that Unix sockets are actually a more
> complex mechanism than pipes.

With Unix sockets, once the connection is made, it's all in-memory
networking. I suspect (but don't know) that named pipes require the
data to pass through the filesystem for each write.

But I could be completely wrong, don't take my word for it.

--
-v System76 Thelio Mega v1.1 x86_64 NVIDIA RTX 3090 Ti
OS: Linux 6.12.0 Release: Mint 21.3 Mem: 258G
"The way to a man's heart is through the left ventricle."

On 22 Nov 2024 06:09:05 GMT, vallor wrote:

> Doesn't the named pipe connection work through the filesystem code? That
> could add overhead.

No. The only thing that exists in the filesystem is the “special file”
entry in the directory. Opening that triggers special-case processing in
the kernel that creates the usual pipe buffering/synchronization
structures (or links up with existing structures created by some prior
opening of the same special file, perhaps by a different process), not
dependent on any filesystem.

I just tried creating a C program to do speed tests on data transfers
through pipes and socket pairs between processes. I am currently setting
the counter to 10 gigabytes, and transferring that amount of data (using
whichever mechanism) only takes a couple of seconds on my system.

So the idea that pipes are somehow not suited to large data transfers is
patently nonsense.

> Can't use named pipes on just any filesystem -- won't work on NFS for
> example, unless I'm mistaken.

Hard to believe NFS could stuff that up, but there you go ...

On 11/21/24 4:56 PM, Lawrence D'Oliveiro wrote:
> On Thu, 21 Nov 2024 10:12:18 -0500, Phillip Frabott wrote:
>
>> On 11/21/2024 02:22, Lawrence D'Oliveiro wrote:
>>> On Thu, 21 Nov 2024 02:05:46 -0500, Phillip Frabott wrote:
>>>
>>>> On 11/19/2024 19:23, Lawrence D'Oliveiro wrote:
>>>>
>>>>> On Mon, 18 Nov 2024 09:45:07 -0500, Phillip Frabott wrote:
>>>>>
>>>>>> I think the point that is being made by calling pipes a "temp files"
>>>>>> is that they are not persistent.
>>>>>
>>>>> Named pipes can indeed be persistent.
>>>>
>>>> Sure, but then your just creating a file with all the limitations that
>>>> come from that.
>>>
>>> Not at all. It still has the same synchronization behaviour.
>>>
>>>> IPC only benefits you when you use unnamed or traditional pipes
>>>> (performance and resources).
>>>
>>> Certainly not.
>>
>> I guess it just depends on what you are doing.
>
> No it doesn’t. Named or not, pipes are pipes.
>
>> And in perspective, most pipes are generally used for small amounts of
>> data ...
>
> I have used them to transfer quite large amounts, quickly and reliably.

Yep, WILL work. No question.

The question is HOW MUCH should you intend to send back
and forth using pipes (or any other method) between the
parent and children.

IMHO, keeping it 'minimal' is Better Programming ... but
others may have different visions/needs/sensibilities.

If you build things that need TOO much data passed
around - do you REALLY want a parent/child sort of
paradigm at all ? "One Big Pgm With Lots Of Globals"
seems more sensible.

On Fri, 22 Nov 2024 01:44:35 -0500, 186282@ud0s4.net wrote:

> On 11/21/24 4:56 PM, Lawrence D'Oliveiro wrote:
>
>> I have used them to transfer quite large amounts, quickly and reliably.
>
> Yep, WILL work. No question.
>
> The question is HOW MUCH should you intend to send back
> and forth using pipes (or any other method) between the
> parent and children.

How about 10 gigabytes, which I was able to transfer in two seconds? Is
that “too much” for you?

On Fri, 22 Nov 2024 06:37:06 -0000 (UTC), Lawrence D'Oliveiro
<ldo@nz.invalid> wrote in <vhp8qi$12m83$2@dont-email.me>:

> On 22 Nov 2024 06:09:05 GMT, vallor wrote:
>
>> Doesn't the named pipe connection work through the filesystem code? That
>> could add overhead.
>
> No. The only thing that exists in the filesystem is the “special file”
> entry in the directory. Opening that triggers special-case processing in
> the kernel that creates the usual pipe buffering/synchronization
> structures (or links up with existing structures created by some prior
> opening of the same special file, perhaps by a different process), not
> dependent on any filesystem.
>
> I just tried creating a C program to do speed tests on data transfers
> through pipes and socket pairs between processes. I am currently setting
> the counter to 10 gigabytes, and transferring that amount of data (using
> whichever mechanism) only takes a couple of seconds on my system.
>
> So the idea that pipes are somehow not suited to large data transfers is
> patently nonsense.
>
>> Can't use named pipes on just any filesystem -- won't work on NFS for
>> example, unless I'm mistaken.
>
> Hard to believe NFS could stuff that up, but there you go ...

Just tested NFS, and named pipes work there.

$ time -p ( dd if=/dev/zero of=test count=$[1024*1024] ) & cat test > /dev/null
[1] 38859
1048576+0 records in
1048576+0 records out
536870912 bytes (537 MB, 512 MiB) copied, 0.918945 s, 584 MB/s
real 0.92
user 0.16
sys 0.76

NFS vers 4.1.

--
-v System76 Thelio Mega v1.1 x86_64 NVIDIA RTX 3090 Ti
OS: Linux 6.12.0 Release: Mint 21.3 Mem: 258G
"Diagonally parked in a parallel universe."

On 22 Nov 2024 07:02:47 GMT, vallor <vallor@cultnix.org> wrote in
<lqaoknF8btnU2@mid.individual.net>:

> On Fri, 22 Nov 2024 06:37:06 -0000 (UTC), Lawrence D'Oliveiro
> <ldo@nz.invalid> wrote in <vhp8qi$12m83$2@dont-email.me>:
>
>> On 22 Nov 2024 06:09:05 GMT, vallor wrote:
>>
>>> Doesn't the named pipe connection work through the filesystem code?
That
>>> could add overhead.
>>
>> No. The only thing that exists in the filesystem is the “special file”
>> entry in the directory. Opening that triggers special-case processing
in
>> the kernel that creates the usual pipe buffering/synchronization
>> structures (or links up with existing structures created by some prior
>> opening of the same special file, perhaps by a different process), not
>> dependent on any filesystem.
>>
>> I just tried creating a C program to do speed tests on data transfers
>> through pipes and socket pairs between processes. I am currently
setting
>> the counter to 10 gigabytes, and transferring that amount of data
(using
>> whichever mechanism) only takes a couple of seconds on my system.
>>
>> So the idea that pipes are somehow not suited to large data transfers
is
>> patently nonsense.
>>
>>> Can't use named pipes on just any filesystem -- won't work on NFS for
>>> example, unless I'm mistaken.
>>
>> Hard to believe NFS could stuff that up, but there you go ...
>
> Just tested NFS, and named pipes work there.
>
> $ time -p ( dd if=/dev/zero of=test count=$[1024*1024] ) & cat test > /
dev/null
> [1] 38859
> 1048576+0 records in
> 1048576+0 records out
> 536870912 bytes (537 MB, 512 MiB) copied, 0.918945 s, 584 MB/s
> real 0.92
> user 0.16
> sys 0.76
>
> NFS vers 4.1.

$ nc -l -U -N /tmp/socket > /dev/null & time -p ( dd if=/dev/zero
count=$[1024*1024*2] | nc -U -N /tmp/socket )
[1] 40284
2097152+0 records in
2097152+0 records out
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 2.03617 s, 527 MB/s
real 2.03
user 0.47
sys 3.60
[1]+ Done nc -l -U -N /tmp/socket > /dev/null

However, the speed appears to be limited by dd in my examples -- setting a
block size to fill the pipe/packets seems to increase throughput:

$ nc -l -U -N /tmp/socket > /dev/null & time -p ( dd if=/dev/zero
count=$[1024*1024*4] bs=1024 | nc -U -N /tmp/socket > /dev/null )
[1] 41764
4194304+0 records in
4194304+0 records out
4294967296 bytes (4.3 GB, 4.0 GiB) copied, 4.02026 s, 1.1 GB/s
real 4.02
user 0.89
sys 7.11

$ time -p ( dd if=/dev/zero of=test count=$[1024*1024*4] bs=$[8*512]) &
cat test > /dev/null
[1] 41282
4194304+0 records in
4194304+0 records out
17179869184 bytes (17 GB, 16 GiB) copied, 4.43357 s, 3.9 GB/s
real 4.43
user 0.54
sys 3.88

$ ulimit -p
8 (pipesize in 512-byte blocks)

(Now I'm off to find out the MTU for Unix sockets...)

--
-v System76 Thelio Mega v1.1 x86_64 NVIDIA RTX 3090 Ti
OS: Linux 6.12.0 Release: Mint 21.3 Mem: 258G
""This is a job for.. AACK! WAAUGHHH!! ...someone else." - Calvin"

On 22 Nov 2024 07:29:16 GMT, vallor <vallor@cultnix.org> wrote in
<lqaq6cF8btnU3@mid.individual.net>:

> However, the speed appears to be limited by dd in my examples -- setting
> a block size to fill the pipe/packets seems to increase throughput:
>
> $ nc -l -U -N /tmp/socket > /dev/null & time -p ( dd if=/dev/zero
> count=$[1024*1024*4] bs=1024 | nc -U -N /tmp/socket > /dev/null )

Realized the bottleneck would be the pipe between dd and nc, so wrote
a program to connect to /tmp/socket and spew data at it -- it sends
46950 212992-byte buffers (9999974400 bytes) in 2.41 seconds.
(4149366971 bytes/second, or 4.1GB/s).

(The default "MTU" for a Linux Unix socket connection
is 212992 bytes. Default pipe size is 8*512 bytes.)

--
-v System76 Thelio Mega v1.1 x86_64 NVIDIA RTX 3090 Ti
OS: Linux 6.12.0 Release: Mint 21.3 Mem: 258G
"Profanity is the one language all programmers know best."

On 22 Nov 2024 08:38:20 GMT, vallor wrote:

> ... or 4.1GB/s).

That’s about what I was getting, for both a pipe (unnamed) and a
socketpair.