Glassthought

Pipe Operator

2 min read

# Example of piping
some_foo | tee /tmp/out

Notes

  • When you pipe you create a separate process.

More in depth notes

Return codes

Capturing regular return code won't work

Regular Return Code Does Not Work with Pipe

foo_unhappy(){
  echo "foo_unhappy"
  return 1
}
 
main() {
  foo_unhappy | tee /tmp/out
  echo "foo_unhappy | tee /tmp/out. Return code: $?"
}
 
main "${@}" || exit 1
foo_unhappy
foo_unhappy | tee /tmp/out. Return code: 0

To properly verify

To properly verify look at:

Link to original

Gotta use ${PIPESTATUS[0]}, to capture return code of first command in a pipe.

${PIPESTATUS[0]} will capture the return code of first command in the pipe.

foo_unhappy(){
  echo "foo_unhappy"
  return 1
}
 
foo_happy(){
  echo "foo_happy"
  return 0
}
main() {
  foo_unhappy | tee /tmp/out
  echo "foo_unhappy returned: ${PIPESTATUS[0]}"
 
  foo_happy | tee /tmp/out
  echo "foo_happy returned: ${PIPESTATUS[0]}"
}
 
main "${@}" || exit 1
foo_unhappy
foo_unhappy returned: 1
foo_happy
foo_happy returned: 0
PIPESTATUS array has return codes of commands executed in the pipe

pipestatus_array_has_return_codes_of_commands_in_a_pipe

Source & Pipe

source ./XX.sh With piped output, will not modify ENV variables

Source to Pipe

Piping output of source makes it so ENV variable changes are not reflected in parent process.

When you pipe you create a sub-process. Hence, you env variable changes would not propagate up to parent, since they were only changed in the child process.

Example Code 
# scratch1.sh
main() {
  SOME_ENV="Scratch1 value"
  echo "SOME_ENV: ${SOME_ENV:?} (in Scratch1) (PID: $$, BASHPID: $BASHPID)"
  echo "--------------------------------------------------------------------------------Starting source"
  source "${SCRATCH_SHELL2:?}" | tee /tmp/out
  echo "--------------------------------------------------------------------------------Finished source"
  echo "SOME_ENV: ${SOME_ENV:?} (in Scratch1) (PID: $$, BASHPID: $BASHPID)"
}
 
main "${@}" || exit 1
# scratch2.sh
main() {
  SOME_ENV="Scratch2-value"
  echo "SOME_ENV: ${SOME_ENV:?} (in Scratch2) (PID: $$, BASHPID: $BASHPID)"
}
 
main "${@}" || exit 1
Link to original

Pipes and interrupt

Works for stopping subsequent pipe command 
foo(){
  echo "Interrupting foo"
  sleep 1
  kill -INT -$$
}
 
bar(){
  echo "BAR START"
 
  while read -r line; do
      # Process each line of input here
      echo "Received: $line"
  done
 
  echo "BAR FINISH"
}
 
main() {
  foo | bar
}
 
main "${@}" || exit 1

Bar will not finish as expected

BAR START
Received: Interrupting foo
Does not work from preventing unchained commands 
foo(){
  echo "Interrupting foo (BASHPID: $BASHPID)"
  sleep 1
  kill -INT -$$
}
 
bar(){
  echo "BAR START (BASHPID: $BASHPID)"
 
  while read -r line; do
      # Process each line of input here
      echo "Received: $line"
  done
 
  echo "BAR FINISH"
}
 
main() {
  foo | bar
 
  echo "MAIN FINISH (BASHPID: $BASHPID)"
}
 
main "${@}" || exit 1
BAR START (BASHPID: 48671)
Received: Interrupting foo (BASHPID: 48670)
MAIN FINISH (BASHPID: 48669)

Pipe Status Verification

Pipe Status Verification

status_verification

Pipe and arguments

Pipe & Arguments

Example processing pipe line by line

Test case

foo() {
  if [ -t 0 ]; then
    echo "Called normally"
    echo "Received arg: $1"
  else
    echo.green "Called from a pipeline"
 
    while IFS= read -r line; do
      echo "Processing piped input: $line"
    done
  fi
}
export -f foo
 
 
main() {
  echo "Calling foo() normally"
  foo "bar"
  echo "--------------------------------------------------------------------------------"
  echo "Calling foo() from a pipeline"
  echo "bar" | foo
}
 
main "${@}" || exit 1

Output:

Calling foo() normally
Called normally
Received arg: bar
--------------------------------------------------------------------------------
Calling foo() from a pipeline
Called from a pipeline
Processing piped input: bar
 
real	0m0.016s
user	0m0.013s
sys	0m0.004s
Compress piped input with JQ 
# shellcheck disable=SC2120
bar(){
  if [ -t 0 ]; then
    echo "BAR Called normally: $*"
  else
    echo "BAR Called from a pipeline"
 
    while IFS= read -r line; do
      echo.bold "BAR Processing piped input: $line"
      sleep 2
    done
  fi
}
 
foo() {
  if [ -t 0 ]; then
    echo "${*:?}" | jq -c | bar
  else
    jq -c | bar
  fi
}
export -f foo
 
main() {
  echo.green "$(date.now)"
  echo.jsonl | jq | foo
  echo.green "$(date.now)"
  foo '{"hi":"there"}'
}
 
main "${@}" || exit 1

Output:

2023-12-14T21-17-51PST
BAR Called from a pipeline
BAR Processing piped input: {"title":"title-val-1","note":"note-val-1"}
BAR Processing piped input: {"title":"title-val-2","note":"note-val-2"}
BAR Processing piped input: {"title":"title-val-3","note":"note-val-3"}
2023-12-14T21-17-57PST
BAR Called from a pipeline
BAR Processing piped input: {"hi":"there"}
Link to original

Graph View