tail & tail -F

# in log directory to tail multiple json logs at once :
tail -F -q * | jq -c

tail -F

tail -F -q and piping to other commands (configuring buffering of other commands)

why-tail-F-instead-of-f

If you want to pipe output of tail -F to commands like grep or jq then look into configuring them so they buffer per line rather than holding on to larger buffers, look at the following notes:

• GREP_OPTIONS-using-line-buffered
• buffering-issue

Also make sure to user -q when piping structured logs (like JSON per line logs) to tools like jq.

with: stdbuf -oL

IF you are working with a file that does not end with an empty line, and you are piping the content from tail -F then prefix tail -F with stdbuf -oL.

Otherwise tail -F is NOT going to output the last line from your file. And you could end up with very confusing tailing missing that last event.

Link to original

Link to original

tail -f --pid=$PID_THAT_KEEPS_US_ALIVE

gem

Problem

You want to tail a file UNTIL some process completes (e.g., monitoring a log file while a job runs). If you use tail -f alone, it will block execution indefinitely even after the process finishes.

Solution

tail -f has an additional option: --pid=$PID that makes tail -f only continue tailing while the provided PID is ALIVE. It automatically stops when the provided PID dies.

From man page:

       --pid=PID
              with -f, terminate after process ID, PID dies

Example snapshot:

GT-Sandbox-Snapshot: tail -f --pid focused by itself

Code

#!/usr/bin/env bash
 
# Create a temporary file
LOGFILE=$(mktemp)
echo "Log file: $LOGFILE"
 
# Function that writes to file periodically
writer() {
    local logfile=$1
    for i in {1..5}; do
        echo "$(date '+%H:%M:%S') - Writing entry $i" >> "$logfile"
        sleep 1
    done
    echo "$(date '+%H:%M:%S') - Writer finished!" >> "$logfile"
}
 
echo "Start the writer in background"
writer "$LOGFILE" &
WRITER_PID=$!
 
echo "Writer PID: $WRITER_PID"
echo "Starting tail -f with --pid=$WRITER_PID"
echo "Watch: tail will automatically exit when writer finishes!"
echo "---"
 
# Monitor with tail -f, will exit when WRITER_PID dies
tail -f "$LOGFILE" --pid=$WRITER_PID
 
echo "---"
echo "tail -f has exited because PID $WRITER_PID finished!"
echo "Final log contents:"
cat "$LOGFILE"
 
# Cleanup
rm "$LOGFILE"

Command to reproduce:

gt.sandbox.checkout.commit 8c4144efd7cf0af3fa39 \
&& cd "${GT_SANDBOX_REPO}/bash" \
&& cmd.run.announce "./main.sh"

Recorded output of command:

Log file: /dev/shm/tmp.aIOrgoKpu0
Start the writer in background
Writer PID: 201326
Starting tail -f with --pid=201326
Watch: tail will automatically exit when writer finishes!
---
13:35:16 - Writing entry 1
13:35:17 - Writing entry 2
13:35:18 - Writing entry 3
13:35:19 - Writing entry 4
13:35:20 - Writing entry 5
13:35:21 - Writer finished!
---
tail -f has exited because PID 201326 finished!
Final log contents:
13:35:16 - Writing entry 1
13:35:17 - Writing entry 2
13:35:18 - Writing entry 3
13:35:19 - Writing entry 4
13:35:20 - Writing entry 5
13:35:21 - Writer finished!

GT-Sandbox-Snapshot: With GNU Parallel

Code

#!/usr/bin/env bash
 
 
# Create a temporary joblog
JOBLOG=$(mktemp)
echo "Job log: $JOBLOG"
echo ""
 
# Create a simple task that takes varying time
slow_task() {
    local n=$1
    local duration=$((n % 5 + 1))  # 1-5 seconds
    sleep $duration
    echo "Task $n completed after ${duration}s"
}
 
# Export the function so parallel can use it
export -f slow_task
 
# Start parallel in background with joblog (no -j limit = all at once)
echo "Starting 10 parallel jobs (all at once)..."
parallel --joblog "$JOBLOG" slow_task ::: {1..10} &
PARALLEL_PID=$!
 
echo "Parallel PID: $PARALLEL_PID"
echo "Monitoring joblog with tail -f --pid=$PARALLEL_PID"
echo "tail will automatically exit when all jobs complete!"
echo "---"
 
# Monitor the joblog, will exit when parallel finishes
tail -f "$JOBLOG" --pid=$PARALLEL_PID
 
echo "---"
echo "All parallel jobs complete! tail -f has exited automatically."
echo ""
echo "Final job summary:"
cat "$JOBLOG"
 
# Cleanup
rm "$JOBLOG"

Command to reproduce:

gt.sandbox.checkout.commit ab5ce91584dee641acc1 \
&& cd "${GT_SANDBOX_REPO}/bash" \
&& cmd.run.announce "./main.sh"

Recorded output of command:

Job log: /dev/shm/tmp.4rZVSWBqBT
 
Starting 10 parallel jobs (all at once)...
Parallel PID: 205914
Monitoring joblog with tail -f --pid=205914
tail will automatically exit when all jobs complete!
---
Seq	Host	Starttime	JobRuntime	Send	Receive	Exitval	Signal	Command
Task 5 completed after 1s
5	:	1759261418.856	     1.032	0	26	0	0	slow_task 5
Task 10 completed after 1s
10	:	1759261418.867	     1.028	0	27	0	0	slow_task 10
Task 1 completed after 2s
1	:	1759261418.849	     2.031	0	26	0	0	slow_task 1
Task 6 completed after 2s
6	:	1759261418.859	     2.031	0	26	0	0	slow_task 6
Task 2 completed after 3s
2	:	1759261418.850	     3.035	0	26	0	0	slow_task 2
Task 7 completed after 3s
7	:	1759261418.861	     3.029	0	26	0	0	slow_task 7
Task 3 completed after 4s
3	:	1759261418.852	     4.032	0	26	0	0	slow_task 3
Task 8 completed after 4s
8	:	1759261418.863	     4.029	0	26	0	0	slow_task 8
Task 4 completed after 5s
4	:	1759261418.854	     5.031	0	26	0	0	slow_task 4
Task 9 completed after 5s
9	:	1759261418.865	     5.029	0	26	0	0	slow_task 9
---
All parallel jobs complete! tail -f has exited automatically.
 
Final job summary:
Seq	Host	Starttime	JobRuntime	Send	Receive	Exitval	Signal	Command
5	:	1759261418.856	     1.032	0	26	0	0	slow_task 5
10	:	1759261418.867	     1.028	0	27	0	0	slow_task 10
1	:	1759261418.849	     2.031	0	26	0	0	slow_task 1
6	:	1759261418.859	     2.031	0	26	0	0	slow_task 6
2	:	1759261418.850	     3.035	0	26	0	0	slow_task 2
7	:	1759261418.861	     3.029	0	26	0	0	slow_task 7
3	:	1759261418.852	     4.032	0	26	0	0	slow_task 3
8	:	1759261418.863	     4.029	0	26	0	0	slow_task 8
4	:	1759261418.854	     5.031	0	26	0	0	slow_task 4
9	:	1759261418.865	     5.029	0	26	0	0	slow_task 9

Link to original