Consider this code in a bash script:

(Note: No actual indentation is intended - only shown like this for posting)

someCommand ... << EOF | someOtherCommand
some data
for someCommand
EOF

This should work, right?

In simple cases, it does (seem to) work OK.

However, in the actual real world case, it is more like:

someCommand -x -y "sjdfhk" and more options \
-g and still more options -Q "another option" << EOF |
/usr/lib/ANotherCommand -x "with option" and "more options"
some data
for someCommand
EOF

This time, two things happen:

When edited with GVIM, everything after the line that ends with | is
highlighted as if it was an unterminated string (that is, in a purple/pink
color) and when the above file is dotted, bash complains about "Syntax
error: Unexpected end of file" - as if it never seems the EOF tag.

In the end, I ended up replacing the " << EOF | ANotherCommand ..." construct with:

> >(ANotherCommand ...) << EOF

and all is well. But why? Should this be necessary?
Yes, I know it is hard to debug this w/o specifics, but I really don't
think the specifics matter; they would be hard to reproduce here.

I just want to know if the basic syntax is valid and/or if there is a
better workaround (better than switching to: > >(...))

--
Many North Koreans believe Kim-il-Sung is an "almighty god" who "created
the world" in seven days as a divine spirit millions of years ago, and
came to Earth as a human in 1912 as a messianic figure.

On 31.10.2024 21:44, Kenny McCormack wrote:
> Consider this code in a bash script:
>
> (Note: No actual indentation is intended - only shown like this for posting)
>
> someCommand ... << EOF | someOtherCommand
> some data
> for someCommand
> EOF
>
> This should work, right?
>
> In simple cases, it does (seem to) work OK.
>
> However, in the actual real world case, it is more like:
>
> someCommand -x -y "sjdfhk" and more options \
> -g and still more options -Q "another option" << EOF |
> /usr/lib/ANotherCommand -x "with option" and "more options"
> some data
> for someCommand
> EOF

This complexity is not helping much to see what issue you have also
when mentioning syntax highlighting problems.

All I can say at the moment - and you decide whether that's relevant
for your case - is that, say, instead of syntax

cat \
-n << EOF |
wc -l
some data
for someCommand
EOF

you might rather want

cat \
-n << EOF |
some data
for someCommand
EOF
wc -l

HTH.

Janis

>
> This time, two things happen:
>
> When edited with GVIM, everything after the line that ends with | is
> highlighted as if it was an unterminated string (that is, in a purple/pink
> color) and when the above file is dotted, bash complains about "Syntax
> error: Unexpected end of file" - as if it never seems the EOF tag.
>
> In the end, I ended up replacing the " << EOF | ANotherCommand ..." construct with:
>
> > >(ANotherCommand ...) << EOF
>
> and all is well. But why? Should this be necessary?
> Yes, I know it is hard to debug this w/o specifics, but I really don't
> think the specifics matter; they would be hard to reproduce here.
>
> I just want to know if the basic syntax is valid and/or if there is a
> better workaround (better than switching to: > >(...))
>

gazelle@shell.xmission.com (Kenny McCormack) writes:

> Consider this code in a bash script:
>
> (Note: No actual indentation is intended - only shown like this for posting)
>
> someCommand ... << EOF | someOtherCommand
> some data
> for someCommand
> EOF
>
> This should work, right?
>
> In simple cases, it does (seem to) work OK.
>
> However, in the actual real world case, it is more like:
>
> someCommand -x -y "sjdfhk" and more options \
> -g and still more options -Q "another option" << EOF |

I think you need "| \" at the end of this line. At least that's what I
usually do and it seems to work.

> /usr/lib/ANotherCommand -x "with option" and "more options"
> some data
> for someCommand
> EOF

--
Ben.

In article <87r07v99wd.fsf@bsb.me.uk>, Ben Bacarisse <ben@bsb.me.uk> wrote:
....
>I think you need "| \" at the end of this line. At least that's what I
>usually do and it seems to work.

It is not generally necessary to put a backslash at the end of a line that
ends with | (in shell script).

But it *is* necessary in this special case! So, you get the prize.
Note that this solves it as far as getting bash to be happy with it is
concerned. When I get a chance, I need to see about how VIM feels about it.

Here's test case:

$ nl << EOF | nl
test
this
EOF
1 1 test
2 2 this
$ nl << EOF |
nl
test
this
EOF

-bash5: syntax error: unexpected end of file
Status: 2
$ nl << EOF | \
nl
test
this
EOF
1 1 test
2 2 this
$

--
It's possible that leasing office space to a Starbucks is a greater liability
in today's GOP than is hitting your mother on the head with a hammer.

On 01.11.2024 00:05, Kenny McCormack wrote:
> In article <87r07v99wd.fsf@bsb.me.uk>, Ben Bacarisse <ben@bsb.me.uk> wrote:
> ...
>> I think you need "| \" at the end of this line. At least that's what I
>> usually do and it seems to work.
>
> It is not generally necessary to put a backslash at the end of a line that
> ends with | (in shell script).
>
> But it *is* necessary in this special case!

Unless you put the 'nl' at the "right place"; you can write your
example below as

$ nl << EOF |
test
this
EOF
nl

so the backslash is not "necessary". - As you say, the '|' needs no
[spurious] continuation escape character if you have it at the end
of a command. - After the lines that define the here-doc (for 'nl's
redirection) the pipe command gets continued on the subsequent line,
which is the line after the "EOF".

> So, you get the prize.
> Note that this solves it as far as getting bash to be happy with it is
> concerned. When I get a chance, I need to see about how VIM feels about it.
>
> Here's test case:
>
> $ nl << EOF | nl
> test
> this
> EOF
> 1 1 test
> 2 2 this
> $ nl << EOF |
> nl
> test
> this
> EOF

Here, 'nl' is part of the here-doc data. so the pipe has no process
to feed its data in. You can see that if you add, say, another 'nl'
behind the "EOF" and you'll get
1 1 nl
2 2 test
3 3 this

>
> -bash5: syntax error: unexpected end of file
> Status: 2
> $ nl << EOF | \
> nl

(That's a spurious [unnecessary] escape I'd consider a hack. - But
okay, YMMV.)

Janis

> test
> this
> EOF
> 1 1 test
> 2 2 this
> $
>

In article <vg18jc$2tv5s$1@dont-email.me>,
Janis Papanagnou <janis_papanagnou+ng@hotmail.com> wrote:
....
>Unless you put the 'nl' at the "right place"; you can write your
>example below as
>
>$ nl << EOF |
>test
>this
>EOF
>nl
>
>so the backslash is not "necessary". - As you say, the '|' needs no
>[spurious] continuation escape character if you have it at the end
>of a command. - After the lines that define the here-doc (for 'nl's
>redirection) the pipe command gets continued on the subsequent line,
>which is the line after the "EOF".

Yes, that works, too. Thanks.

Somewhat surprisingly (to me, anyway), I think I actually prefer to stick
with; > >(cmd ...)

That has a certain beauty to it.

--
Reading any post by Fred Hodgin, you're always faced with the choice of:
lunatic, moron, or troll.

I always try to be generous and give benefit of the doubt, by assuming troll.

On 2024-10-31, Kenny McCormack <gazelle@shell.xmission.com> wrote:
> In article <87r07v99wd.fsf@bsb.me.uk>, Ben Bacarisse <ben@bsb.me.uk> wrote:
> ...
>>I think you need "| \" at the end of this line. At least that's what I
>>usually do and it seems to work.
>
> It is not generally necessary to put a backslash at the end of a line that
> ends with | (in shell script).

The << feature works outside of the ordinary shell grammar, fetching
data in a line-oriented way until the delimiter.

If you have multiple occurrences, both their data parts go after
the command:

cmd1 << EOF | cmd2 << EOF | cmd3 << EOF
data1
EOF
data2
EOF
data3
EOF

Actual run with && rather than |:

$ cat << EOF && cat << EOF && cat << EOF
> foo
> EOF
> bar
> EOF
> xyzzy
> EOF
foo
bar
xyzzy

It looks like the implementation of << pulls logical lines (after
backslash folding) during parsing.

If this entire command pipe were parsed first:

cmd1 << EOF | cmd2 << EOF | cmd3 << EOF

and then a second pass took place to fetch the EOF-delimited pieces of
syntax that it requires, then it could contain artbitrary backslahes.

--
TXR Programming Language: http://nongnu.org/txr
Cygnal: Cygwin Native Application Library: http://kylheku.com/cygnal
Mastodon: @Kazinator@mstdn.ca

Il Fri, 1 Nov 2024 09:01:15 -0000 (UTC), Kenny McCormack ha scritto:

> I think I actually prefer to stick with; > >(cmd ...)

Note that

cat << EOF | grep . ; echo end

and also

(cat | grep .) << EOF ; echo end

produce the same output:
---------------------------------------------------------
lem@biggy:~$ cat << EOF | grep . ; echo end
start
EOF
start
end
lem@biggy:~$
---------------------------------------------------------
lem@biggy:~$ (cat | grep .) << EOF ; echo end
> start
> EOF
start
end
lem@biggy:~$
---------------------------------------------------------

Instead with "cat << EOF > >(grep .) ; echo end" the output is
unpredictable. You can very well get, as usual:
---------------------------------------------------------
lem@biggy:~$ cat << EOF > >(grep .) ; echo end
start
EOF
start
end
lem@biggy:~$
---------------------------------------------------------
OR maybe:
---------------------------------------------------------
lem@biggy:~$ cat << EOF > >(grep .) ; echo end
start
EOF
end
start
lem@biggy:~$
---------------------------------------------------------
OR even, most of the time:
---------------------------------------------------------
lem@biggy:~$ cat << EOF > >(grep .) ; echo end
start
EOF
end
lem@biggy:~$ start [you press ENTER here and get back an empty prompt]

lem@biggy:~$
---------------------------------------------------------

So sometimes it's better to be careful. :)
--
Bye, Lem

gazelle@shell.xmission.com (Kenny McCormack):
> Consider this code in a bash script:
>
>
> (Note: No actual indentation is intended - only shown like this
> for posting)
>
>
> someCommand ... << EOF | someOtherCommand
> some data
> for someCommand
> EOF
>
> This should work, right?
>
>
> In simple cases, it does (seem to) work OK.
>
>
> However, in the actual real world case, it is more like:
>
>
> someCommand -x -y "sjdfhk" and more options \
> -g and still more options -Q "another option" << EOF |
> /usr/lib/ANotherCommand -x "with option" and "more options"
> some data
> for someCommand
> EOF
>
> This time, two things happen:
>
>
> When edited with GVIM, everything after the line that ends with
> | is highlighted as if it was an unterminated string (that is,
> in a purple/pink color) and when the above file is dotted, bash
> complains about "Syntax error: Unexpected end of file" - as if
> it never seems the EOF tag.
>

I don't think so.  It doesn't see any command after the pipe
symbol (because the next and following lines are the here
document).  That's why GVIM shows the color used to indicate
strings and here documents.

> In the end, I ended up replacing the " << EOF | ANotherCommand
> ..." construct with:
>
>
> > >(ANotherCommand ...) << EOF
>
> and all is well. But why? Should this be necessary?
>

It isn't necessary, because there are at least three
alternatives.

> Yes, I know it is hard to debug this w/o specifics, but I really
> don't think the specifics matter; they would be hard to
> reproduce here.
>
>
> I just want to know if the basic syntax is valid and/or if there is a
> better workaround (better than switching to: > >(...))
>

Either use a backslash for unfolding a folded line (as Ben
showed) or put the part of the command line following the pipe
symbol ("|") after the EOF of the here document (as Janis showed)
or embrace the simple command and its here document by "{ ... ;
}":

{
someCommand -x -y "sjdfhk" and more options \
-g and still more options -Q "another option" << EOF
some data
for someCommand
EOF
} |
/usr/lib/ANotherCommand -x "with option" and "more options"

On 03.11.2024 21:37, Helmut Waitzmann wrote:
> gazelle@shell.xmission.com (Kenny McCormack):
>>
>> I just want to know if the basic syntax is valid and/or if there is a
>> better workaround (better than switching to: > >(...))
>
> Either use a backslash for unfolding a folded line (as Ben
> showed) or put the part of the command line following the pipe
> symbol ("|") after the EOF of the here document (as Janis showed)
> or embrace the simple command and its here document by "{ ... ;
> }":
>
>
> {
> someCommand -x -y "sjdfhk" and more options \
> -g and still more options -Q "another option" << EOF
> some data
> for someCommand
> EOF
> } |
> /usr/lib/ANotherCommand -x "with option" and "more options"
>

In this vein (i.e. structuring the code) another option is to put
all these long commands with all their options into functions, so
that the complex command calls fit (with the pipes, and without
escapes) on a single line

f_someCommand <<EOF | f_anotherCommand
...
EOF

(This could make the code also better readable and maintainable.)

Janis