/////////////////////
//////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////
# /the/tech/blog
Solutions for FreeBSD web and application servers using PHP, mySQL, Apache and more.
Adam Strohl
Sunday, September 12th 2010

Adam StrohlSOLVED: Output of Multiple Greps Are Delayed or Missing (No Output) When Tailing a File

I recently ran into an issue when running the output of tail -F file.log through a series of multiple grep instances.

One level of grep worked fine and I saw the output in real time. However, once I added a second level, I would get nothing for minutes after, then sudden output (at first I thought my regex syntax was off).

What I discovered was happening is that grep (at least on FreeBSD) buffers its output by default. Multiple pipes/calls means multiple buffers and that causes the delay. Adding the --line-buffered option to grep forces grep to output each line immediately (and thus passing it on to the next grep process in the pipe):

# tail -F haystack.log | grep --line-buffered "needle" |
     grep -v --line-buffered " junk "

Now you can filter your log files in real time!

As a side note I have used grep for years and only recently ran into this issue, specifically with Apache web server log files. There might be some interaction with how Apache buffers the writing of log files, or how tail -F works.

loading icon

Fetching vote status, one moment ...

Add a Comment:

* Name:
* Email address:
Verification purposes only. Your email address will not be shown publicly, transfered, or sold. We don't spam and take data privacy seriously.
* Comment:
icon

Related Experience

Take a look at these related items:

UNIX® Experience
FreeBSD Server Administration
Apache Server Support

icon

Disclaimer

Any problems and their solutions discussed here are designed to be examples and may not exactly match your situation.

No warranty is provided or implied with these articles, if it's critical contact us: we support and guarantee all of our work.