Tag Archives: bash

How to send passive checks to nagios real life example:

Posted by on January 13, 2016 No comments

First of all – why you need to use passive checks in nagios.
It`s useful for large systems, nagios will not wait for connect timeout during telecom issues.
And it`s easy to configure.

Our case (large social network).
Need to check number of unsubscribers. If no “unsubscribe” letters for 1 hour – something goes wrong… FBL list not working and we need Alert. If we will not process FBL letters for several hours, email providers rise our SPAM rating.

How to fetch letters (I use ruby Imap) – topic for another article :).

1. Nagios Check code:

# cat /home/scripts/fbl.sh
#!/bin/bash

NUM=`/usr/bin/psql -t -h 1.1.1.1 -p 5450 -U cron_user  base3 -c "select count(1) from email_stop_list where (esl_created BETWEEN current_timestamp - interval '1 hour' and current_timestamp) and esl_reason ~ '^fbl'"`

if [ $NUM -eq 0 ]; then
        echo -e "nest\tunsubscribe_fbl\t3\tNo_Unsubscribe"  | /home/scripts/send_nsca -H 2.2.2.2 -p 5667 -c /etc/send_nsca.conf
 else
    echo -e "nest\tunsubscribe_fbl\t0\t$NUM unsubscribes last houer"  | /home/scripts/send_nsca -H 2.2.2.2 -p 5667 -c /etc/send_nsca.conf
 fi

2. Code for send_nsca

Plugin Return Code Service State Host State
0 OK UP
1 WARNING UP or DOWN/UNREACHABLE*
2 CRITICAL DOWN/UNREACHABLE
3 UNKNOWN DOWN/UNREACHABLE

3. Nginx service config

# cat nest.cfg
define service{
  use                            generic-service-template-passive
  host_name                       nest
  service_description             unsubscribe_fbl
  freshness_threshold             3600
  check_command                   volatile_no_information
  contact_groups                  nagios-wheel,nagios-wheel-smsmail
}

4. Service template

define service {
    use                             generic-service-template
    name                            generic-service-template-passive
    active_checks_enabled           0
    passive_checks_enabled          1
    obsess_over_service             0
    flap_detection_enabled          0
    event_handler_enabled           1
    failure_prediction_enabled      1
    is_volatile                     1
    register                        0
    check_period                    24x7
    max_check_attempts              1
    normal_check_interval           5
    retry_check_interval            2
    check_freshness                 1
    freshness_threshold             90000
    contact_groups                  nagios-wheel
    check_command                   volatile_no_information
    notifications_enabled           1
    notification_interval           15
    notification_period             24x7
    notification_options            w,u,c,r
    process_perf_data               1
    retain_status_information       1
    retain_nonstatus_information    1
}

How to tar.gz yesterday logs (some etl magic)

Posted by on September 21, 2015 No comments

Task: need to tar yesteday logs in one file and gzip it.
Little bash code, just to save my time in future.

#!/bin/bash

src='/var/spool/etl/archive'

dt=`date --date="1 day ago" +"%Y-%m-%d"`
#create empty tar archive
tar cvf $src/$dt.tar --files-from /dev/null

for i in `ls -1 $src/*$dt* | grep -v gz | grep -v tar`; do
  tar -rf $src/$dt.tar $i
  rm -f $i
done
gzip $src/$dt.tar

video sound track merger

Posted by on September 18, 2015 No comments

Few “years” ago I made part of small promo project for Nestle Russia as subcontractor.
It was promo action, website with some videos. Kids make sound track and my task was to merge user sound track and original video soundtrack.
It`s really easy to do with ffmpeg or mencoder.

BTW: ffmeg much better it works o.k. with aac codec and mp4 container.

Code is VERY VERY dirty, we had absolutely no time, but it can be useful to someone. And I save it “just to remember”.

#!/bin/bash

( #start subprocess
  # Wait for lock on /var/lock/.merger-1.lock (fd 200) for 10 seconds
  flock -x -w 3 200
  if [ "$?" != "0" ]; then echo Cannot lock!; exit 1; fi
  echo $$>>/var/lock/.merger-1.lock #for backward lockdir compatibility, notice this command is executed AFTER command bottom  ) 200>/var/lock/.myscript-1.exclusivelock.


sourcevideo="/var/www/kinder_prod/sourcevideo"
sourceaudio="/var/www/kinder_prod/audioupload"
targetdir="/var/www/kinder_prod/processedvideo"
processedaudio="/var/www/kinder_prod/processedaudio"

while true; do

if [ "$(ls -A $sourceaudio)" ]; then

  for i in `ls -1 $sourceaudio/*.wav | xargs -n1 basename`; do
  videoid=`echo $i | awk -F"--" '{print $1}'`
  audioid=`echo $i | awk -F"--" '{print $2}' | awk -F"." '{print $1}'`

  sox $sourceaudio/$i /tmp1/$i rate 44100; mv /tmp1/$i $sourceaudio/$i; chown milkslice:milkslice $sourceaudio/$i || exit 1

  sox -m $sourcevideo/$videoid.mp3 $sourceaudio/$i /tmp1/$videoid--$audioid.mp3 && \
  ffmpeg -y -i /tmp1/$videoid--$audioid.mp3 -strict experimental -acodec aac -bsf:a aac_adtstoasc /tmp1/$videoid--$audioid.aac && \
    ffmpeg -y -i /tmp1/$videoid--$audioid.aac -i $sourcevideo/$videoid.mp4 -bsf:a aac_adtstoasc -preset ultrafast -c copy $targetdir/$videoid--$audioid.mp4 || exit 1
#   mencoder -of lavf -lavfopts format=mp4 -oac copy  -fafmttag 0x706D  \
#-audiofile /tmp1/$videoid--$audioid.aac  -ovc copy $sourcevideo/$videoid.mp4 -o $targetdir/$videoid--$audioid.mp4 || exit 1
    chown milkslice:milkslice $targetdir/$videoid--$audioid.mp4
    mv -f $sourceaudio/$i $processedaudio
    rm /tmp1/$videoid--$audioid.mp3
        rm /tmp1/$videoid--$audioid.aac

    done

fi

sleep 1;
done

) 200>/var/lock/.merger-1.lock   #exit subprocess

FLOCKEXIT=$?  #save exitcode status

exit $FLOCKEXIT

And run screen with script. (alternative to upstart)

/usr/bin/screen -dm bash -c 'cd /root/merger-prod; /root/merger-prod/merger-prod.sh'

How to delete files without big iowait

Posted by on July 23, 2015 No comments

I know 2 ways, tested in high loaded production.

if scheduler support ionice (on some systems makes LA)

 # ionice -c 3 nice -n 20 find  /DIRECTORY -type f -delete

Just ajust sleep time, according to your system LA

while true; do find /DIRECTORY/ -type f -print  -delete -quit; sleep 0.01; done

iptables SNAT vs MASQUERADE

Posted by on November 25, 2014 No comments

What is a difference and why should we use SNAT instead of MASQUERADE.

According to official documentation:

There is a specialized case of Source NAT called masquerading: it should only be used for dynamically-assigned IP addresses, such as standard dialups (for static IP addresses, use SNAT above).

With SNAT, the kernel’s connection tracking keeps track of all the connections when the interface is taken down and brought back up. For the MASQUERADE target connection will be lost.

With MASQUERADE some issues can occur if your have more than one ip on outgoing interface.

With MASQUERADE kernel determine nat outgoing ip address for every connection (it looks for interface IP) it`s rather expensive operation.

But in 99.99% cases MASQUERADE is o.k.

I Use following iptables construction to nat rear outgoing SMTP connections. (postfix started at physical server, and lxc containers relay mail to base system via ssmtp or nullmailer).

/sbin/iptables -t nat -A POSTROUTING -s 10.2.1.254/32  -o eth0 -j MASQUERADE

It`s universal chain, works great at number of servers, and you should not determine outgoing interface address. (as for -j SNAT –to-source X.X.X.X)

my OsX bash .profile

Posted by on November 3, 2014 No comments

Just to remember. I add some locale settings to avoid ????? instead of letters in console Unicode programms (such as MC)

# MacPorts Installer addition on 2013-01-22_at_14:43:51: adding an appropriate PATH variable for use with MacPorts.
export PATH=/opt/local/bin:/opt/local/sbin:$PATH
# Finished adapting your PATH environment variable for use with MacPorts.

HISTCONTROL=ignoredups:ignorespace

# for setting history length see HISTSIZE and HISTFILESIZE in bash
HISTSIZE=2500
HISTFILESIZE=5000

export LANG=en_US.UTF-8
export LANGUAGE=en_US.UTF-8
export LC_MESSAGES=POSIX
export LC_ALL=en_US.UTF-8

export editor=/usr/bin/vim

export PS1='\[\e]0;\u@\h: \w\a\]\[\033[01;31m\]\u\[\033[01;33m\]@\[\033[01;36m\]\h \[\033[01;33m\]\w \[\033[01;35m\]\$ \[\033[00m\]'

export force_color_prompt=yes
export HISTTIMEFORMAT="%h/%d - %H:%M:%S "

    #color ls
    alias ls='ls -G'
    alias grep='grep --color=auto'
    alias fgrep='fgrep --color=auto'
    alias egrep='egrep --color=auto'
 
    alias ll='ls -alF'
    alias la='ls -A'
    alias l='ls -CF'

linux how to create a service

Posted by on September 24, 2014 No comments

First of all read http://upstart.ubuntu.com/getting-started.html upstart is beautiful 🙂 If your linux supports it.
But in some systems you should use old init style scripts 🙁

cp /etc/init.d/sceleton /etc/init.d/rec-runner
and edit it.

/etc/init.d/rec-runner
Read more »

How to test cdn delivery speed via curl

Posted by on June 8, 2014 No comments

Our company use our own CDN based on nginx caching. 7 high loaded (40 000 RPS per server) servers in 2 datacenters.
And periodically I observer some deviations in delivery time. from 0.15 to 7.5 or even 30 seconds.
We have nginx SLA module + Graphics and monitoring. But I need to test all servers for anomaly delivery time.

#!/bin/bash

for l in ip1.x.x.x \
         ip2.x.x.x \
         ....
         ipN.x.x.x; do

echo $l;

    for i in {1..1024}; do
    curl -s -w "%{time_total} -- %{time_connect}\n" -o /dev/null --resolve it.randomthemes.com:443:$l https://it.randomthemes.com/favicon.ico >> ./$l.txt
    done

done

Then analyse ipN.x.x.x.txt any way you like.

cat | sort -n | tail -n 25
etc.

Have a nice day. I really like curl and hope this will help someone.

ext4 perfomance tuning

Posted by on May 17, 2014 No comments

I use following mount options.
In some projects it gives significant performance boost.

errors=remount-ro – need for hardware problem case. Because if disc remains mounted, further writing attempts can deadly damage file system. And one more case – easy monitoring. Just check via zabbix or nagios that you have no ro file system.

noatime, nodiratime – not fix access time. Double check that your applications doesn`t need this.

discard – use trim for SSD drive. In case SATA or SAS this option ignored by system.
commit, nobarrier – dangerous in case of power outage. But for my infrastructure o.k.

ext4 errors=remount-ro,noatime,nodiratime,commit=100,discard,nobarrier

And SED for fstab fixing (I use puppet, chef, fabric).

sed -r -i 's/ext4\s+defaults/ext4 errors=remount-ro,noatime,nodiratime,commit=100,discard,nobarrier/' /etc/fstab

Mail 550 filter

Posted by on September 25, 2013 No comments

If you make a project with huge amount of email notification, you MUST control number of 550 reply from mail servers. Because if you skip this step, and continue sending to deleted mail boxes, big mail providers such as gmail.com, mail.ru, mail.ua, etc. will ban you domain at 0.5 – 1% “user unknown” reply.
So mail.log parsing is only solution.
In our project we add bad email addresses to database table (we use postgress)

1. make database replace rule, if email already added, (email is primary key) it is fastest way to prevent errors on INSERT duplicate email addresses.
Read more »