apache logs – fluentd – mongodb – Hizli , sipsak loglari toplamaca …

Senaryomuz ;

• Elimizde bir adet sanal makinemiz mevcut, uzerine ubuntu kurulmus , apache yuklenmis , free bir web site template ile icerik yaratilmis durumda.
• Yaratilan apache access log’larini hem aggregate edip hemde tek bir yere toplayacagiz
• Hem store olarak hemde uzerinde sorgulama yapmak icin MongoDB kullanacagiz

 

---

 

Hizlica tek node mongodb kuralim 

Kullanilacak isletim sistemi

noroot@mongodb01single:~$ lsb_release -a

No LSB modules are available.

Distributor ID: Ubuntu

Description: Ubuntu 14.04.3 LTS

Release: 14.04

Codename: trusty

noroot@mongodb01single:~$ sudo apt-key adv –keyserver hkp://keyserver.ubuntu.com:80 –recv 7F0CEB10

[sudo] password for noroot:

Executing: gpg –ignore-time-conflict –no-options –no-default-keyring –homedir /tmp/tmp.Ijdtvnfxbw –no-auto-check-trustdb –trust-model always –keyring /etc/apt/trusted.gpg –primary-keyring /etc/apt/trusted.gpg –keyserver hkp://keyserver.ubuntu.com:80 –recv 7F0CEB10

gpg: requesting key 7F0CEB10 from hkp server keyserver.ubuntu.com

gpg: key 7F0CEB10: public key “Richard Kreuter <richard@10gen.com>” imported

gpg: Total number processed: 1

noroot@mongodb01single:~$ echo “deb http://repo.mongodb.org/apt/ubuntu trusty/mongodb-org/3.2 multiverse” | sudo tee /etc/apt/sources.list.d/mongodb-org-3.2.list

deb http://repo.mongodb.org/apt/ubuntu trusty/mongodb-org/3.2 multiverse

Tekrar guncelleyelim

noroot@mongodb01single:~$ sudo apt-get update

Kuralim gitsin 

noroot@mongodb01single:~$ sudo apt-get install mongodb-org

Varsayili olarak 127.0.0.1 uzerinden ve 27017 portunu dinler , asagidaki satiri devre disi birakin

#  bindIp: 127.0.0.1

sudo vi /etc/mongod.conf

Env degerlerine dikkat mutlaka UTF8 olsun , mongo yazili shell’e dusmeden once bunu yapin

noroot@mongodb01single:~$ cat /etc/environment

PATH=”/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games”

LANG=en_US.utf-8

LC_ALL=en_US.utf-8

Mongo servisine bir restart 

noroot@mongodb01single:~$ sudo service mongod restart

mongod stop/waiting

mongod start/running, process 1283

---

 

Fluentd Konfigrasyonu 

 

Oncelikle td-agent.conf dosyasini editleyelim

noroot@mongodb01single:~$ sudo vi /etc/td-agent/td-agent.conf

Dosyanin en sonuna asagidaki satirlari ekleyin …

<source>

#Bildigimiz tail ile okuayacagiz apache logunu

type tail

#Bu arada eski log dosyaslari okunmayacak, sadece yeni gelenler

format apache2

#Apache log dosyasi ve yolu

path /var/log/apache2/access.log

#Log dosyasini okurken nerede kaldigini bilmek icin pos file okunuyor

pos_file /var/log/td-agent/apache2.access_log.pos

#Routing i belirtmek icin etiketliyoruz

tag apache.access

</source>

<match apache.access>

#Mongo ya atacagiz

type mongo

#Mongodb db adi + collection adi

#Bu arada default kurulumda kullanici adi ve sifre geregi yok

database apachedb

collection accesslogs

#Mongodb dinledigi port ve ip adresi

host 10.111.21.161

port 27017

#Mongoya datayi atam suresi

flush_interval 10s

#Zaman etiketinin oldugunu garanti edelim

include_time_key true

</match>

 

ve servisi restart edin

noroot@apache01fluentd:~$ sudo service td-agent restart

* Restarting td-agent  td-agent

Logdan kontrol edelim , calisiyor mu cidde ? 

noroot@apache01fluentd:~$ tail -f /var/log/td-agent/td-agent

td-agent.log     td-agent-ui.log

noroot@apache01fluentd:~$ tail -f /var/log/td-agent/td-agent.log 

collection accesslogs

host 10.111.21.161

port 27017

flush_interval 10s

include_time_key true

</match>

</ROOT>

2015-12-24 15:05:15 +0200 [info]: listening fluent socket on 0.0.0.0:24224

2015-12-24 15:05:15 +0200 [info]: listening dRuby uri=”druby://127.0.0.1:24230″ object=”Engine”

2015-12-24 15:05:15 +0200 [info]: following tail of /var/log/apache2/access.log

Simdi Mongo tarafini kontrol edelim 

mongo shell >

> show dbs

apachedb  0.000GB

local     0.000GB

> use apachedb

switched to db apachedb

> show collections

accesslogs

> db.accesslogs.findOne()

{

“_id” : ObjectId(“567beb25cec70e0835000001”),

“host” : “10.111.21.160”,

“user” : null,

“method” : “GET”,

“path” : “/”,

“code” : 200,

“size” : 23479,

“referer” : null,

“agent” : “ApacheBench/2.3”,

“time” : ISODate(“2015-12-24T12:54:58Z”)

}

>

Baslangic icin simdilik bu …

VM