Pipe Operator
# Example of piping
some_foo | tee /tmp/out
Notes
- When you pipe you create a separate process.
More in depth notes
Return codes
Capturing regular return code won't work
From Regular Return Code Does Not Work with Pipe
Go to text →
foo_unhappy(){
echo "foo_unhappy"
return 1
}
main() {
foo_unhappy | tee /tmp/out
echo "foo_unhappy | tee /tmp/out. Return code: $?"
}
main "${@}" || exit 1
foo_unhappy
foo_unhappy | tee /tmp/out. Return code: 0
To properly verify
To properly verify look at:
Gotta use ${PIPESTATUS[0]}, to capture return code of first command in a pipe.
${PIPESTATUS[0]}
will capture the return code of first command in the pipe.
foo_unhappy(){
echo "foo_unhappy"
return 1
}
foo_happy(){
echo "foo_happy"
return 0
}
main() {
foo_unhappy | tee /tmp/out
echo "foo_unhappy returned: ${PIPESTATUS[0]}"
foo_happy | tee /tmp/out
echo "foo_happy returned: ${PIPESTATUS[0]}"
}
main "${@}" || exit 1
foo_unhappy
foo_unhappy returned: 1
foo_happy
foo_happy returned: 0
PIPESTATUS array has return codes of commands executed in the pipe
No note with name tech.bash.lang.pipe.PIPESTATUS_array_has_return_codes_of_commands_in_a_pipe found in cache during parsing.
Source & Pipe
source ./XX.sh With piped output, will not modify ENV variables
From Source to Pipe
Go to text →
Piping output of source makes it so ENV variable changes are not reflected in parent process.
When you pipe you create a sub-process. Hence, you env variable changes would not propagate up to parent, since they were only changed in the child process.
Example Code
# scratch1.sh
main() {
SOME_ENV="Scratch1 value"
echo "SOME_ENV: ${SOME_ENV:?} (in Scratch1) (PID: $$, BASHPID: $BASHPID)"
echo "--------------------------------------------------------------------------------Starting source"
source "${SCRATCH_SHELL2:?}" | tee /tmp/out
echo "--------------------------------------------------------------------------------Finished source"
echo "SOME_ENV: ${SOME_ENV:?} (in Scratch1) (PID: $$, BASHPID: $BASHPID)"
}
main "${@}" || exit 1
# scratch2.sh
main() {
SOME_ENV="Scratch2-value"
echo "SOME_ENV: ${SOME_ENV:?} (in Scratch2) (PID: $$, BASHPID: $BASHPID)"
}
main "${@}" || exit 1
Pipes and interrupt
Works for stopping subsequent pipe command
foo(){
echo "Interrupting foo"
sleep 1
kill -INT -$$
}
bar(){
echo "BAR START"
while read -r line; do
# Process each line of input here
echo "Received: $line"
done
echo "BAR FINISH"
}
main() {
foo | bar
}
main "${@}" || exit 1
Bar will not finish as expected
BAR START
Received: Interrupting foo
Does not work from preventing unchained commands
foo(){
echo "Interrupting foo (BASHPID: $BASHPID)"
sleep 1
kill -INT -$$
}
bar(){
echo "BAR START (BASHPID: $BASHPID)"
while read -r line; do
# Process each line of input here
echo "Received: $line"
done
echo "BAR FINISH"
}
main() {
foo | bar
echo "MAIN FINISH (BASHPID: $BASHPID)"
}
main "${@}" || exit 1
BAR START (BASHPID: 48671)
Received: Interrupting foo (BASHPID: 48670)
MAIN FINISH (BASHPID: 48669)
Pipe Status Verification
Pipe Status Verification
Pipe and arguments
From Pipe & Arguments
Go to text →
Example processing pipe line by line
Test case
foo() {
if [ -t 0 ]; then
echo "Called normally"
echo "Received arg: $1"
else
echo.green "Called from a pipeline"
while IFS= read -r line; do
echo "Processing piped input: $line"
done
fi
}
export -f foo
main() {
echo "Calling foo() normally"
foo "bar"
echo "--------------------------------------------------------------------------------"
echo "Calling foo() from a pipeline"
echo "bar" | foo
}
main "${@}" || exit 1
Output:
Calling foo() normally
Called normally
Received arg: bar
--------------------------------------------------------------------------------
Calling foo() from a pipeline
Called from a pipeline
Processing piped input: bar
real 0m0.016s
user 0m0.013s
sys 0m0.004s
Compress piped input with JQ
# shellcheck disable=SC2120
bar(){
if [ -t 0 ]; then
echo "BAR Called normally: $*"
else
echo "BAR Called from a pipeline"
while IFS= read -r line; do
echo.bold "BAR Processing piped input: $line"
sleep 2
done
fi
}
foo() {
if [ -t 0 ]; then
echo "${*:?}" | jq -c | bar
else
jq -c | bar
fi
}
export -f foo
main() {
echo.green "$(date.now)"
echo.jsonl | jq | foo
echo.green "$(date.now)"
foo '{"hi":"there"}'
}
main "${@}" || exit 1
Output:
2023-12-14T21-17-51PST
BAR Called from a pipeline
BAR Processing piped input: {"title":"title-val-1","note":"note-val-1"}
BAR Processing piped input: {"title":"title-val-2","note":"note-val-2"}
BAR Processing piped input: {"title":"title-val-3","note":"note-val-3"}
2023-12-14T21-17-57PST
BAR Called from a pipeline
BAR Processing piped input: {"hi":"there"}
Children