Pipe Operator

# Example of piping
some_foo | tee /tmp/out

Notes

When you pipe you create a separate process.

More in depth notes

Return codes

Capturing regular return code won't work

From Regular Return Code Does Not Work with Pipe

Go to text →

foo_unhappy(){
  echo "foo_unhappy"
  return 1
}

main() {
  foo_unhappy | tee /tmp/out
  echo "foo_unhappy | tee /tmp/out. Return code: $?"
}

main "${@}" || exit 1

foo_unhappy
foo_unhappy | tee /tmp/out. Return code: 0

To properly verify

To properly verify look at:

Gotta use ${PIPESTATUS[0]}, to capture return code of first command in a pipe.

${PIPESTATUS[0]} will capture the return code of first command in the pipe.

foo_unhappy(){
  echo "foo_unhappy"
  return 1
}

foo_happy(){
  echo "foo_happy"
  return 0
}
main() {
  foo_unhappy | tee /tmp/out
  echo "foo_unhappy returned: ${PIPESTATUS[0]}"

  foo_happy | tee /tmp/out
  echo "foo_happy returned: ${PIPESTATUS[0]}"
}

main "${@}" || exit 1

foo_unhappy
foo_unhappy returned: 1
foo_happy
foo_happy returned: 0

PIPESTATUS array has return codes of commands executed in the pipe

From PIPESTATUS_array_has_return_codes_of_commands_in_a_pipe

Go to text →

foo_unhappy(){
  local input_data=$(cat)
  echo -n "foo_unhappy(${input_data})"
  return 1
}

foo_happy(){
  echo -n "foo_happy($(cat))"
  return 0
}

main() {
  echo "start it off" | foo_unhappy | foo_unhappy | foo_happy | foo_unhappy | tee /tmp/out
  echo "PIPESTATUS: ${PIPESTATUS[@]}"
}

main "${@}" || exit 1

New liens added in the output for clarity

foo_unhappy(foo_happy(foo_unhappy(foo_unhappy(start it off))))

PIPESTATUS: 0 1 1 0 1 0

PIPESTATUS corresponds the the status of each elements within the pipe (starting with echo in the above example).

Source & Pipe

source ./XX.sh With piped output, will not modify ENV variables

From Source to Pipe

Go to text →

Piping output of source makes it so ENV variable changes are not reflected in parent process.

When you pipe you create a sub-process. Hence, you env variable changes would not propagate up to parent, since they were only changed in the child process.

Example Code

# scratch1.sh
main() {
  SOME_ENV="Scratch1 value"
  echo "SOME_ENV: ${SOME_ENV:?} (in Scratch1) (PID: $$, BASHPID: $BASHPID)"
  echo "--------------------------------------------------------------------------------Starting source"
  source "${SCRATCH_SHELL2:?}" | tee /tmp/out
  echo "--------------------------------------------------------------------------------Finished source"
  echo "SOME_ENV: ${SOME_ENV:?} (in Scratch1) (PID: $$, BASHPID: $BASHPID)"
}

main "${@}" || exit 1

# scratch2.sh
main() {
  SOME_ENV="Scratch2-value"
  echo "SOME_ENV: ${SOME_ENV:?} (in Scratch2) (PID: $$, BASHPID: $BASHPID)"
}

main "${@}" || exit 1

Pipes and interrupt

Works for stopping subsequent pipe command

foo(){
  echo "Interrupting foo"
  sleep 1
  kill -INT -$$
}

bar(){
  echo "BAR START"

  while read -r line; do
      # Process each line of input here
      echo "Received: $line"
  done

  echo "BAR FINISH"
}

main() {
  foo | bar
}

main "${@}" || exit 1

Bar will not finish as expected

BAR START
Received: Interrupting foo

Does not work from preventing unchained commands

foo(){
  echo "Interrupting foo (BASHPID: $BASHPID)"
  sleep 1
  kill -INT -$$
}

bar(){
  echo "BAR START (BASHPID: $BASHPID)"

  while read -r line; do
      # Process each line of input here
      echo "Received: $line"
  done

  echo "BAR FINISH"
}

main() {
  foo | bar

  echo "MAIN FINISH (BASHPID: $BASHPID)"
}

main "${@}" || exit 1

BAR START (BASHPID: 48671)
Received: Interrupting foo (BASHPID: 48670)
MAIN FINISH (BASHPID: 48669)

Pipe Status Verification

Status_verification

Pipe and arguments

From Pipe & Arguments

Go to text →

Example processing pipe line by line

Test case

foo() {
  if [ -t 0 ]; then
    echo "Called normally"
    echo "Received arg: $1"
  else
    echo.green "Called from a pipeline"

    while IFS= read -r line; do
      echo "Processing piped input: $line"
    done
  fi
}
export -f foo


main() {
  echo "Calling foo() normally"
  foo "bar"
  echo "--------------------------------------------------------------------------------"
  echo "Calling foo() from a pipeline"
  echo "bar" | foo
}

main "${@}" || exit 1

Output:

Calling foo() normally
Called normally
Received arg: bar
--------------------------------------------------------------------------------
Calling foo() from a pipeline
Called from a pipeline
Processing piped input: bar

real	0m0.016s
user	0m0.013s
sys	0m0.004s

Compress piped input with JQ

# shellcheck disable=SC2120
bar(){
  if [ -t 0 ]; then
    echo "BAR Called normally: $*"
  else
    echo "BAR Called from a pipeline"

    while IFS= read -r line; do
      echo.bold "BAR Processing piped input: $line"
      sleep 2
    done
  fi
}

foo() {
  if [ -t 0 ]; then
    echo "${*:?}" | jq -c | bar
  else
    jq -c | bar
  fi
}
export -f foo

main() {
  echo.green "$(date.now)"
  echo.jsonl | jq | foo
  echo.green "$(date.now)"
  foo '{"hi":"there"}'
}

main "${@}" || exit 1

Output:

2023-12-14T21-17-51PST
BAR Called from a pipeline
BAR Processing piped input: {"title":"title-val-1","note":"note-val-1"}
BAR Processing piped input: {"title":"title-val-2","note":"note-val-2"}
BAR Processing piped input: {"title":"title-val-3","note":"note-val-3"}
2023-12-14T21-17-57PST
BAR Called from a pipeline
BAR Processing piped input: {"hi":"there"}

Children

Notes