Category Archives: Linux - Page 3

some qute bash compare

Posted by on February 19, 2014 No comments

Task – need to generate routes file for openvpn from list of our networks.
Check, if file different from current route script, replace one and do some action. This task occurs very often.
Code is very simple, so:

cat /etc/ipfw.list | awk '{print "push "route " $1 "" "  }' > /root/test1;
if [ `diff /root/test /root/test1 | wc -l ` -eq 0  ];  
  then  
    echo "no difference";
  else echo "differ";
    rm -f /root/test; mv /root/test1 /root/test;
fi

Generate Unique Request ID nginx

Posted by on January 21, 2014 1 comment

Task – Need to add unique ID to each user request. External nginx module such as request ID is very unstable, so I write small perl script to generate UUID and add it to header.
nginx embedded perl is extremely fast, and works very well in high loaded production systems.

Required packages:

aptitude install libossp-uuid-perl

/etc/nginx/nginx.conf

http {
...
perl_require "Data/UUID.pm";
perl_set $uuid 'sub {

  $ug = new Data::UUID;
  $str = $ug->create_str();
  return $str;
               }';
... }

Location config:

    location ~ /data/(.+) {
...
...
            proxy_set_header    X-Request-Id    $uuid;
...
}

Mail 550 filter

Posted by on September 25, 2013 No comments

If you make a project with huge amount of email notification, you MUST control number of 550 reply from mail servers. Because if you skip this step, and continue sending to deleted mail boxes, big mail providers such as gmail.com, mail.ru, mail.ua, etc. will ban you domain at 0.5 – 1% “user unknown” reply.
So mail.log parsing is only solution.
In our project we add bad email addresses to database table (we use postgress)

1. make database replace rule, if email already added, (email is primary key) it is fastest way to prevent errors on INSERT duplicate email addresses.
Read more »

Mysql can do partitioning on the fly :)

Posted by on July 23, 2013 No comments

Mysql tuning in action…
Yesterday me, and our development team made some tuning of one old project we have couple of tables with 1-10 million records.
It`s NOT BIGDATA, but application makes huge writes to this table.
Only current date records, but you understand partitioning affects index size.
Table can be partitioned by date field. So how to do it:

  1. make sure that dt not null.
  2. recreate primary key. Field for partition should be part of primary key.
  3. create partition for old years and partition for every month.

Read more »

Mysql slave lag monitoring

Posted by on May 16, 2013 2 comments

Everybody know

SHOW SLAVE STATUS;

Also everybody know that ‘Seconds_Behind_Master’ shows difference in seconds between the slave SQL thread and the slave I/O thread.

Sometime it shows nonsense, and if you build monitoring It is not good practice to use ‘Seconds_Behind_Master’
Example from real life:
If replication becomes stalled due to connectivity problems, Seconds_Behind_Master shows 0 while replica is far away from master, changing timeout values not help. 🙁 i mean:

slave_net_timeout=300

So we implement following monitoring – every 3 second write current timestamp at master.
and check replica delay.
Or You can use percona heartbeat, It do almost the same.
Read more »

How to capture bad email addresses in mass mail.

Posted by on April 16, 2013 No comments

We have huge project with a great number of registered users. They receive notification via email, when actions occur. (new comment, gift, some other activity). Project targeting – Russia and former USSR In Russia some free email hosting providers like mail.ru delete user account after 2-3 years of inactivity.
So we have now 2-3000000 users with bad email address.
How to find them and remove from mailing list:

According to RFC 5321 (smtp rfc) in case of wrong email in rcpt to, server should return 550 no such user.
Read more »

How to make BIG partitions in linux

Posted by on January 10, 2013 No comments

Your cannot create partitions larger then 4Tb using fdisk.
Your should use parted and gpt as partition type.
Read more »

Mysql check auto_increment values

Posted by on January 10, 2013 No comments

We get old project, with braindead architecture.
Database structure provides surprises regularly. Yesterday we had a problem with maximum integer value reached at one column.

‘id’ at one table was auto_increment and “INT” not UNSIGNED INT or BIG INT, so after reaching id value 2147483647 application stop working, I change type to UNSIGNED BIGINT.

ALTER TABLE ad_campaigns_rates MODIFY id BIGINT UNSIGNED  not null auto_increment, AUTO_INCREMENT = 2147483648;

Read more »

Squid with simple authentication

Posted by on November 15, 2012 No comments

Typical task:
We order server at India, for development needs. Need to setup web proxy with authentication. Use squid, Luke!

#htpasswd -c  /etc/squid/passwd username
#htpasswd  /etc/squid/passwd

squid.conf

forwarded_for off #don`t show user real IP at HTTP headers
auth_param basic program /usr/lib64/squid/ncsa_auth /etc/squid/passwd
<span class="zbench-more-link"> <a href="https://it.randomthemes.com/2012/11/15/squid-with-simple-authentication/#more-193" class="more-link">Read more &raquo;</a></span>

qmail maillists and Jira messages delivery issues.

Posted by on October 29, 2012 No comments

We had interesting issue with jira and qmail.
No mail comes to mail list group.

Not trivial email routing probles, message for mail list comes to ezmlm, but ezmlm rejects messages with tag 

Precedence: bulk

Read more »