Process Larger Log files Simple using Perl Script [ TimesTen In-Memory Database ]

 Using PERL script how can we process large log files easily. In this example i will explain you how we can process Oracle times ten in memory log files.We can make simple modification for that script and we can use to parse large files like size 5 GB. We can use this script for process Mysql, Sql, Psql  and all other database also.

Before starting writing script i will explain you what is Times Ten DB. 

TimesTen In-Memory Database

        Oracle TimesTen In-Memory Database (TimesTen) is a full-featured, memory-optimized, relational database with persistence and recoverability. It provides applications with the instant responsiveness and very high throughput required by database-intensive applications. Deployed in the application tier, TimesTen operates on databases that fit entirely in physical memory (RAM). Applications access the TimesTen database using standard SQL interfaces. For customers with existing application data residing on the Oracle Database, TimesTen is deployed as an in-memory cache database with automatic data synchronization between TimesTen and the Oracle Database.

      It increase speed of data access and performance for accessing data from db. 

      We can use the following places to boost the speed, performance and delay latency in application. How is TimesTen used by real-life customers? TimesTen is deployed by thousands of customers spanning industries in communications, financial services, web applications, travel logistics, gaming, and more.

Script Code for process

  

  By default times ten db create a log file with the names of

Keyagent.log 

We can use this script to ptocess other db [mysql , oracle , psql .. etc]. 

 
package AgentParser;

use 5.22.1;
use strict;
use warnings;
use Errstate::Return;
require Exporter;
our @ISA = qw(Exporter);
our %EXPORT_TAGS = ( 'all' => [ qw() ] );
our @EXPORT_OK = ( @{ $EXPORT_TAGS{'all'} } );
our @EXPORT = qw(Agentlog);
our $VERSION = '0.01';

BEGIN
{
   if(! -d "/var/lib/perl_lib/" )
   {
       print "\nDependent directory /var/lib/perl_lib/ not found!!!\n\n";
       exit 1;
   }
   unshift(@INC,"/var/lib/perl_lib");

}

my ($LogFile,$outfile,@stringgrep);
my $loglevel = "DEBUG";
my $logger = Return->new();
$logger->setlogfile($LogFile,'a');
$logger->disable_console();

sub _shiftlog($)
{
        $LogFile = shift || "Keyagent.log";
        return $LogFile;
}

sub _shiftout($)
{
        $outfile = shift || "Agentlog_Count.txt";
        return $outfile;
}

sub CleanUP($)
{
    my $Flag = shift || '1';
    $logger->closelog();
    exit($Flag);
}

sub untaring($)
{
    $logger->info("untaring() started...");
    my $file = shift || 'agent.log.1';
    if($file !~ m/gz/)
    {
        print "$file - $!\n";
        $logger->error("GUNZIP:file $file not available");
        CleanUP(1);
    }
 system("gunzip $file"); 
    $file=~s/\.gz//;
    if($? ne 0)
 {
        $logger->error("GUNZIP: not working");
     CleanUP(1);   
    }
    $logger->info("untaring() Completed...");
 parsing($file);
}

sub parsing($)
{
    my $c = 0;
    $logger->info("parsing() started...");
    my $parsefile = shift || 'agent.log.1';
    if(! -e $parsefile)
    {
        print "$parsefile - $!\n";
        CleanUP(1);
    }
    open(OUTFILE,">>$outfile") or die("FILE:not able to open file $!\n");
 my @array = `grep -oiP '(\\d{4}\-\\d{2}\-\\d{2})' $parsefile | head -n1  && grep -oiP '\\d{2}\:\\d{2}\:\\d{2}' $parsefile | head -n1`;
    my @end=`grep -oiP '(\\d{4}\-\\d{2}\-\\d{2})' $parsefile | head -n1  && grep -oiP '\\d{2}\:\\d{2}\:\\d{2}' $parsefile | tail -n1`;
    chomp(@end,@array,$parsefile);
    print OUTFILE "\n---------FILE: $parsefile---------------\n";
    print  OUTFILE "LOG START TIME: $array[0] $array[1]\nLOG END TIME: $end[0] $end[1]\n";
 foreach my $key (@stringgrep)
 {
        my $count = `grep -o "$key" $parsefile |wc -l`;
        if($count > 0)
        {
            print OUTFILE "\nKEYWORD: \"$key\"\nNO.Of.Counts:$count";
      my @pattern=`grep "$key" $parsefile`;
            LP:if(!@pattern)
            {
                $logger->error("KEYWORD: \"$key\"  not exist in file: \[$parsefile\]");
                next;
            }
         print OUTFILE @pattern;
        }
        else
        {
            $c++;
            goto LP;
        }
    }
            print OUTFILE "$c LINES NOT MATCHED CHECK LOG FILE 'Keyagent.log' \n" if($c);
    
    $logger->info("Parsing() Completed...");
    $logger->info("----------------------\n");

    print "INFO: Parsing Completed check logfile: Keyagent.log and output file Agentlog_Count.txt\n";
}

sub Agentlog
{
 _shiftlog(@_);
 _shiftout(@_); 
 opendir(DIR, ".") || die "can't opendir $!";
 my @dirmatch = grep { /agent(\d+)*(\.)*(log)*(\d+)*(\.)*(gz)*|agent(\.)*(log)*(\.)*(\d+)*(\.)*(gz)*/ && -f "/home/kodiak/$_" }   readdir(DIR);
 closedir DIR;

 @stringgrep=("Starting NuoDB agent","Running as Broker", "Setting up Raft server","Raft server started","Converting to Leader","Peer joined","Node joined","Peer left","Node left","Node state changed to ACTIVE","Node state changed to RUNNING","has become active", "was previously inactive for","agent is shutting down","ShutdownHook called","has become inactive");

 foreach my $filename (@dirmatch)
 {
      chomp($filename)||$filename=~s/\n//;
      $filename = $filename =~ m/\.gz/ ? untaring($filename) : parsing($filename);
 }
}


1;

__END__

=head1 NAME

AgentParser - To Parse the Timesten Default Agentlog files under Linux/Unix system.

=head1 SYNOPSIS

  use Timten::AgentParser;
  $array=AgentParser->Agentlog("Logfile","outputfile");

=head1 DESCRIPTION

To Parse the Timesten Default Agentlog files under Linux/Unix system.

=head2 EXPORT

use Timten::AgentParser;
  $array=AgentParser->Agentlog("Logfile","outputfile");

=head1 SEE ALSO

=head1 AUTHOR

Name :Kaavannan, K<br>
E-Mail: kaavannaniisc@gmail.com

=head1 COPYRIGHT AND LICENSE

Copyright (C) 2016 by Kaavannan K

This library is free software; you can redistribute it and/or modify
it under the same terms as Perl itself, either Perl version 5.22.1 or,
at your option, any later version of Perl 5 you may have available.


=cut

For further more details please visit the link http://search.cpan.org/~kaavannan/Timten-AgentParser/.

I felt it is very useful to process easily.