Using PERL script how can we process
large log files easily. In this example i will explain you how we can
process Oracle times ten in memory log files.We can make simple
modification for that script and we can use to parse large files like
size 5 GB. We can use this script for process Mysql, Sql, Psql and all
other database also.
Before starting writing script i will explain you what is Times Ten DB.
TimesTen In-Memory Database
Oracle TimesTen In-Memory Database (TimesTen) is a full-featured, memory-optimized, relational database
with persistence and recoverability. It provides applications with the
instant responsiveness and very high throughput required by
database-intensive applications. Deployed in the application tier,
TimesTen operates on databases that fit entirely in physical memory
(RAM). Applications access the TimesTen database using standard SQL
interfaces. For customers with existing application data residing on the
Oracle Database, TimesTen is deployed as an in-memory cache database
with automatic data synchronization between TimesTen and the Oracle
Database.
It increase speed of data access and performance for accessing data from db.
We can use the following places to boost the speed, performance
and delay latency in application. How is TimesTen used by real-life
customers? TimesTen is deployed by
thousands of customers spanning industries in communications, financial
services, web applications, travel logistics, gaming, and more.
Script Code for process
By default times ten db create a log file with the names of
Keyagent.log
We can use this script to ptocess other db [mysql , oracle , psql .. etc].
package AgentParser;
use 5.22.1;
use strict;
use warnings;
use Errstate::Return;
require Exporter;
our @ISA = qw(Exporter);
our %EXPORT_TAGS = ( 'all' => [ qw() ] );
our @EXPORT_OK = ( @{ $EXPORT_TAGS{'all'} } );
our @EXPORT = qw(Agentlog);
our $VERSION = '0.01';
BEGIN
{
if(! -d "/var/lib/perl_lib/" )
{
print "\nDependent directory /var/lib/perl_lib/ not found!!!\n\n";
exit 1;
}
unshift(@INC,"/var/lib/perl_lib");
}
my ($LogFile,$outfile,@stringgrep);
my $loglevel = "DEBUG";
my $logger = Return->new();
$logger->setlogfile($LogFile,'a');
$logger->disable_console();
sub _shiftlog($)
{
$LogFile = shift || "Keyagent.log";
return $LogFile;
}
sub _shiftout($)
{
$outfile = shift || "Agentlog_Count.txt";
return $outfile;
}
sub CleanUP($)
{
my $Flag = shift || '1';
$logger->closelog();
exit($Flag);
}
sub untaring($)
{
$logger->info("untaring() started...");
my $file = shift || 'agent.log.1';
if($file !~ m/gz/)
{
print "$file - $!\n";
$logger->error("GUNZIP:file $file not available");
CleanUP(1);
}
system("gunzip $file");
$file=~s/\.gz//;
if($? ne 0)
{
$logger->error("GUNZIP: not working");
CleanUP(1);
}
$logger->info("untaring() Completed...");
parsing($file);
}
sub parsing($)
{
my $c = 0;
$logger->info("parsing() started...");
my $parsefile = shift || 'agent.log.1';
if(! -e $parsefile)
{
print "$parsefile - $!\n";
CleanUP(1);
}
open(OUTFILE,">>$outfile") or die("FILE:not able to open file $!\n");
my @array = `grep -oiP '(\\d{4}\-\\d{2}\-\\d{2})' $parsefile | head -n1 && grep -oiP '\\d{2}\:\\d{2}\:\\d{2}' $parsefile | head -n1`;
my @end=`grep -oiP '(\\d{4}\-\\d{2}\-\\d{2})' $parsefile | head -n1 && grep -oiP '\\d{2}\:\\d{2}\:\\d{2}' $parsefile | tail -n1`;
chomp(@end,@array,$parsefile);
print OUTFILE "\n---------FILE: $parsefile---------------\n";
print OUTFILE "LOG START TIME: $array[0] $array[1]\nLOG END TIME: $end[0] $end[1]\n";
foreach my $key (@stringgrep)
{
my $count = `grep -o "$key" $parsefile |wc -l`;
if($count > 0)
{
print OUTFILE "\nKEYWORD: \"$key\"\nNO.Of.Counts:$count";
my @pattern=`grep "$key" $parsefile`;
LP:if(!@pattern)
{
$logger->error("KEYWORD: \"$key\" not exist in file: \[$parsefile\]");
next;
}
print OUTFILE @pattern;
}
else
{
$c++;
goto LP;
}
}
print OUTFILE "$c LINES NOT MATCHED CHECK LOG FILE 'Keyagent.log' \n" if($c);
$logger->info("Parsing() Completed...");
$logger->info("----------------------\n");
print "INFO: Parsing Completed check logfile: Keyagent.log and output file Agentlog_Count.txt\n";
}
sub Agentlog
{
_shiftlog(@_);
_shiftout(@_);
opendir(DIR, ".") || die "can't opendir $!";
my @dirmatch = grep { /agent(\d+)*(\.)*(log)*(\d+)*(\.)*(gz)*|agent(\.)*(log)*(\.)*(\d+)*(\.)*(gz)*/ && -f "/home/kodiak/$_" } readdir(DIR);
closedir DIR;
@stringgrep=("Starting NuoDB agent","Running as Broker", "Setting up Raft server","Raft server started","Converting to Leader","Peer joined","Node joined","Peer left","Node left","Node state changed to ACTIVE","Node state changed to RUNNING","has become active", "was previously inactive for","agent is shutting down","ShutdownHook called","has become inactive");
foreach my $filename (@dirmatch)
{
chomp($filename)||$filename=~s/\n//;
$filename = $filename =~ m/\.gz/ ? untaring($filename) : parsing($filename);
}
}
1;
__END__
=head1 NAME
AgentParser - To Parse the Timesten Default Agentlog files under Linux/Unix system.
=head1 SYNOPSIS
use Timten::AgentParser;
$array=AgentParser->Agentlog("Logfile","outputfile");
=head1 DESCRIPTION
To Parse the Timesten Default Agentlog files under Linux/Unix system.
=head2 EXPORT
use Timten::AgentParser;
$array=AgentParser->Agentlog("Logfile","outputfile");
=head1 SEE ALSO
=head1 AUTHOR
Name :Kaavannan, K<br>
E-Mail: kaavannaniisc@gmail.com
=head1 COPYRIGHT AND LICENSE
Copyright (C) 2016 by Kaavannan K
This library is free software; you can redistribute it and/or modify
it under the same terms as Perl itself, either Perl version 5.22.1 or,
at your option, any later version of Perl 5 you may have available.
=cut
For further more details please visit the link
http://search.cpan.org/~kaavannan/Timten-AgentParser/.
I felt it is very useful to process easily.
