Tag Archives: Linux - Page 2

linux how to create a service

Posted by on September 24, 2014 No comments

First of all read http://upstart.ubuntu.com/getting-started.html upstart is beautiful 🙂 If your linux supports it.
But in some systems you should use old init style scripts 🙁

cp /etc/init.d/sceleton /etc/init.d/rec-runner
and edit it.

/etc/init.d/rec-runner
Read more »

rpaf for apache

Posted by on September 11, 2014 No comments

Today I scan it.randomthemes.com with acunetix.
And suddenly found than
/server-status enabled with public access… WTF

/etc/apache2/mods-enabled # cat ./status.conf

<Location /server-status>
    SetHandler server-status
    Order deny,allow
    Deny from all
    Allow from 127.0.0.1 ::1
</Location>

And part of nginx config

  proxy_set_header   Host             $host;
  proxy_set_header   X-Real-IP        $remote_addr;
  proxy_set_header   X-Forwarded-For  $proxy_add_x_forwarded_for;

Headers looks o.k…

DAMN! I JUST FORGET TO INSTALL RPAF FOR APACHE:(

About apache rpaf
It changes the remote address of the client visible to other Apache modules when two conditions are satisfied. First condition is that the remote client is actually a proxy that is defined in httpd.conf. Secondly if there is an incoming X-Forwarded-For header and the proxy is in it’s list of known proxies it takes the last IP from the incoming X-Forwarded-For header and changes the remote address of the client in the request structure. It also takes the incoming X-Host header and updates the virtualhost settings accordingly. For Apache2 mod_proxy it takes the X-Forwared-Host header and updates the virtualhosts

How to test cdn delivery speed via curl

Posted by on June 8, 2014 No comments

Our company use our own CDN based on nginx caching. 7 high loaded (40 000 RPS per server) servers in 2 datacenters.
And periodically I observer some deviations in delivery time. from 0.15 to 7.5 or even 30 seconds.
We have nginx SLA module + Graphics and monitoring. But I need to test all servers for anomaly delivery time.

#!/bin/bash

for l in ip1.x.x.x \
         ip2.x.x.x \
         ....
         ipN.x.x.x; do

echo $l;

    for i in {1..1024}; do
    curl -s -w "%{time_total} -- %{time_connect}\n" -o /dev/null --resolve it.randomthemes.com:443:$l https://it.randomthemes.com/favicon.ico >> ./$l.txt
    done

done

Then analyse ipN.x.x.x.txt any way you like.

cat | sort -n | tail -n 25
etc.

Have a nice day. I really like curl and hope this will help someone.

How to secure wipe file system

Posted by on May 17, 2014 No comments

Before you cancel rented dedicated server, it`s good practice to secure wipe disc drives. Reboot to recovery console, and:
Use shred, Luke!

shred -n 0 -f -v -z /dev/sda

ext4 perfomance tuning

Posted by on May 17, 2014 No comments

I use following mount options.
In some projects it gives significant performance boost.

errors=remount-ro – need for hardware problem case. Because if disc remains mounted, further writing attempts can deadly damage file system. And one more case – easy monitoring. Just check via zabbix or nagios that you have no ro file system.

noatime, nodiratime – not fix access time. Double check that your applications doesn`t need this.

discard – use trim for SSD drive. In case SATA or SAS this option ignored by system.
commit, nobarrier – dangerous in case of power outage. But for my infrastructure o.k.

ext4 errors=remount-ro,noatime,nodiratime,commit=100,discard,nobarrier

And SED for fstab fixing (I use puppet, chef, fabric).

sed -r -i 's/ext4\s+defaults/ext4 errors=remount-ro,noatime,nodiratime,commit=100,discard,nobarrier/' /etc/fstab

Generate Unique Request ID nginx

Posted by on January 21, 2014 1 comment

Task – Need to add unique ID to each user request. External nginx module such as request ID is very unstable, so I write small perl script to generate UUID and add it to header.
nginx embedded perl is extremely fast, and works very well in high loaded production systems.

Required packages:

aptitude install libossp-uuid-perl

/etc/nginx/nginx.conf

http {
...
perl_require "Data/UUID.pm";
perl_set $uuid 'sub {

  $ug = new Data::UUID;
  $str = $ug->create_str();
  return $str;
               }';
... }

Location config:

    location ~ /data/(.+) {
...
...
            proxy_set_header    X-Request-Id    $uuid;
...
}

github backup script

Posted by on August 28, 2013 No comments

Our projects repos hosting on github.
How I make backup (clone all repos using cron + github api) probably there is other way how to make backup, but I don`t find this way.

#!/usr/bin/ruby
require 'rubygems'
require 'octokit'


$git_binary = '/usr/bin/git'
$git_login = "LOGIN"
$git_password = "PASSWORD"
$clone_path = "/home/git"


client = Octokit::Client.new(:login => "#{$git_login}", :password => "#{$git_password}")

repos = client.organization_repositories("ORGANIZATION")

system ("rm -rf #{$clone_path}/ORGANIZATION")

repos.each do |num|

system ("#{$git_binary} clone --mirror --recursive  https://#{$git_login}:#{$git_password}@github.com/#{num.full_name} #{$clone_path}/#{num.full_name}")

end

Mysql slave lag monitoring

Posted by on May 16, 2013 2 comments

Everybody know

SHOW SLAVE STATUS;

Also everybody know that ‘Seconds_Behind_Master’ shows difference in seconds between the slave SQL thread and the slave I/O thread.

Sometime it shows nonsense, and if you build monitoring It is not good practice to use ‘Seconds_Behind_Master’
Example from real life:
If replication becomes stalled due to connectivity problems, Seconds_Behind_Master shows 0 while replica is far away from master, changing timeout values not help. 🙁 i mean:

slave_net_timeout=300

So we implement following monitoring – every 3 second write current timestamp at master.
and check replica delay.
Or You can use percona heartbeat, It do almost the same.
Read more »

How to capture bad email addresses in mass mail.

Posted by on April 16, 2013 No comments

We have huge project with a great number of registered users. They receive notification via email, when actions occur. (new comment, gift, some other activity). Project targeting – Russia and former USSR In Russia some free email hosting providers like mail.ru delete user account after 2-3 years of inactivity.
So we have now 2-3000000 users with bad email address.
How to find them and remove from mailing list:

According to RFC 5321 (smtp rfc) in case of wrong email in rcpt to, server should return 550 no such user.
Read more »

How to make BIG partitions in linux

Posted by on January 10, 2013 No comments

Your cannot create partitions larger then 4Tb using fdisk.
Your should use parted and gpt as partition type.
Read more »