How to find logs that contain certain string in lots of gz/regular files and save it as one/more txt files?

Tags:

Answer: 1

32 hours ago

New question on 22.9.2016, down below!

My path looks like this:

~/Desktop/logs
├── first_folder
|   ├── 11.11 (folder)
│   |   ├── access_log
│   |   ├── access_log.1.gz
│   |   :
│   |   └── access_log.40.gz
|   └── 11.12 (folder)
│       ├── access_log
│       ├── access_log.1.gz
│       :
│       └── access_log.16.gz
├── second_folder (folder)
|   ├── 31.11 (folder)
│   |   ├── access_log
│   |   ├── access_log.1.gz
│   |   :
│   |   └── access_log.20.gz
|   └── 31.15 (folder)
│       ├── access_log
│       ├── access_log.1.gz
│       :
:       └── access_log.38.gz
└── last_folder
    ├── 91.11 (folder)
    |   ├── access_log
    |   ├── access_log.1.gz
    |   :
    |   └── access_log.25.gz
    └── 91.15 (folder)
        ├── access_log
        ├── access_log.1.gz
        :
        └── access_log.30.gz

From all the logs I have to extract the data that contain string: /Jan/2016

Question #1

How can I get all the records from all the files in all folders and save it as single_file.txt

Edit #1

Possible answer:

$ find . -name \*.* | xargs -0 zgrep -E '/Jan/2016' > single_file.txt

File is really really huge, lots of GB.

Output:

single_file.txt
./first_folder/11.11/access_log.9.gz: ... text ...
./first_folder/11.12/access_log.9.gz: ... text ...
./second_folder/31.11/access_log.9.gz: ... text ...
./second_folder/31.11/access_log.9.gz: ... text ...
:

Question #2

How can I get all the records from all the files in all folders and save them individually depending on records I grab from first_folder, second_folder etc like first.txt, second.txt.

Output would look something like:

first.txt
./first_folder/11.11/access_log.9.gz: ... text ...
./first_folder/11.12/access_log.9.gz: ... text ...
:
second.txt
./second_folder/31.11/access_log.9.gz: ... text ...
./second_folder/31.15/access_log.9.gz: ... text ...
:

Question #3

How can I get list of all the files that contain string /Jan/2016?

If I use this command, will it retrieve all the files:

$ find ~/Desktop/logs/ -type f | xargs zgrep -l "/Jan/2016"  

Output:

Terminal
/home/name/Desktop/logs/first_folder/11.11/access_log.9.gz
/home/name/Desktop/logs/first_folder/11.12/access_log.8.gz
/home/name/Desktop/logs/second_folder/31.11/access_log.6.gz
:

Edit #2

@waltinator-s code and @Zanna-s correction:

pushd ~/Desktop/logs
for dir in * ; do
    if [[ -d "$dir" ]] ; then
        outname="$dir.txt"
        find "$dir" -type f -print0 | xargs -0 zgrep -l '/Jan/2016' >"$outname"
    fi
done
popd

will give me structure:

~/Desktop/logs
├── first_folder
|   └── first.txt
├── second_folder
|   └── second.txt
:
└── last_folder
    └── last.txt

where first.txt-last.txt will contain paths to specific files that contain string /Jan/2016.

first.txt
first_folder/11.11/access_log.9.gz
first_folder/11.11/access_log.8.gz
first_folder/11.12/access_log.9.gz
first_folder/11.12/access_log.8.gz

Question #4 (22.9.2016)

I needed to modify /Jan/2016 to certain period of time...for example 1/Nov/2014-31/Apr/2015 so in a code that @Zana provided instead of the /Jan/2016 I used /(Nov|Dec)/2014|/(Jan|Feb|Mar|Apr)/2015. Warning is thrown:

xargs: Warning: a NUL character occurred in the input.  It cannot be passed through in the argument list.  Did you mean to use the --null option?

All results are not returned in created files, althought all files are created.

Added by: Melisa Ryan

Answer: 2

31 hours ago

The modern way to use find, bearing in mind that you will eventually see a filename containing spaces, is with -print0 and xargs -0:

# list all filenames containing '/Jan/2016'
find ~/Desktop/logs -type f -print0 | xargs -0 zgrep -l '/Jan/2016'
# 1. Have all the data from all folders that contain that string under one text file
find ~/Desktop/logs -type f -print0 | xargs -0 zgrep -l '/Jan/2016' >one.text.file
#
# 2. Have data that contains string in a separate text files depending on a folder (example: first folder - first.txt etc)
pushd ~/Desktop/logs
for dir in * ; do
    if [[ -d "$dir" ]] ; then
        outname="$dir.txt"
        find "~/Desktop/logs/$dir" -type f -print0 | xargs -0 zgrep -l '/Jan/2016' >"$outname"
    fi
done
popd
Added by: Robbie Nitzsche MD

Answer: 3

25 hours ago

  1. Have data that contains string in a separate text files depending on a folder (example: first folder - first.txt etc)

You could use a very simple loop to make a text file with the records for that directory inside the directory:

for d in ~/Desktop/logs/* ; do zgrep -E '/Jan/2016' "$d"/* >"$d"/out.txt ; done

Desktop/logs
├── first_folder
│   ├── access_log
│   ├── access_log.gz
│   └── out.txt
└── second_folder
    ├── access_log
    ├── access_log.gz
    └── out.txt

Using this slight adjustment to waltinator's script:

for d in ~/Desktop/logs/* ; do
    if [[ -d "$d" ]] ; then
        outname="$d".txt
        find "$d" -type f -print0 | xargs -0 zgrep -E '/Jan/2016' >"$outname"
    fi
done

will give this structure:

├── first_folder
│   ├── access_log
│   └── access_log.gz
├── first_folder.txt
├── second_folder
│   ├── access_log
│   └── access_log.gz
└── second_folder.txt

Notes

  • for d in ~/Desktop/logs/* ; do loop over the contents of ~/Desktop/logs and do something with them
  • if [[ -d "$d" ]]; then do something only if $d is a directory
  • find "$d" -type f -print0 search inside directories $d for files and output them with the null separator so that we can use
  • xargs -0 construct a command with the output of the previous command as arguments, using the null character as separator (otherwise filenames with spaces break this approach)
  • zgrep searches in zipped/compressed archives
Added by: Lurline Dickinson

Popular Search

A B C D E F G H I J K L M N O P Q R S T U V W X Y Z 1 2 3 4 5 6 7 8 9