bind GENERATE and CDN

In our advertising network we use landing page with great number of images.

And previous team use domain static.OURDOMAIN.net for CDN. But modern browsers open from 4 to 6 simultaneous connections to one domain name. So it takes huge ammount of time to load all page.

i.e. 4 images start loading, browser waits for 1 image load complete, next image loading.
Damn not good and extremely slow :)
so use force Luke:

bind zone:

1
2
3
4
$GENERATE 1-64 static$      IN  A   1.1.1.1     ;s7
$GENERATE 1-64 static$      IN  A   2.2.2.2     ;sgr1
$GENERATE 1-64 static$      IN  A   3.3.3.3     ;sf6
$GENERATE 1-64 static$      IN  A   4.4.4.4     ;sf31

And Use something like http://static.’random(1-64)’.OURDOMAIN.net/IMAGE.PNG in application code.

We significantly speed up page loading. (up to 3 times).

my OsX bash .profile

Just to remember. I add some locale settings to avoid ????? instead of letters in console Unicode programms (such as MC)

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
# MacPorts Installer addition on 2013-01-22_at_14:43:51: adding an appropriate PATH variable for use with MacPorts.
export PATH=/opt/local/bin:/opt/local/sbin:$PATH
# Finished adapting your PATH environment variable for use with MacPorts.

export LANG=en_US.UTF-8
export LANGUAGE=en_US.UTF-8
export LC_MESSAGES=POSIX
export LC_ALL=en_US.UTF-8

export editor=/usr/bin/vim

export PS1='\[\e]0;\u@\h: \w\a\]\[\033[01;31m\]\u\[\033[01;33m\]@\[\033[01;36m\]\h \[\033[01;33m\]\w \[\033[01;35m\]\$ \[\033[00m\]'

export force_color_prompt=yes
export HISTTIMEFORMAT="%h/%d - %H:%M:%S "

Linux How to increase maximum open files for running process.

We use postgres and pgbouncer as kernel DB for one huge project. Some secure requirements:

1
2
3
4
5
6
7
/etc/security/limits.conf
soft nofile 65535
hard nofile 65535

/etc/sysctl.conf
fs.file-max = 1000000
fs.inotify.max_user_watches = 1000000

Don`t forget to remove # at /etc/pam.d/login (remarked by default). And how to change nolimit on the fly.
Read more »

linux how to create a service

First of all read http://upstart.ubuntu.com/getting-started.html upstart is beautiful :) If your linux supports it.
But in some systems you should use old init style scripts :(

cp /etc/init.d/sceleton /etc/init.d/rec-runner
and edit it.

/etc/init.d/rec-runner
Read more »

rpaf for apache

Today I scan it.randomthemes.com with acunetix.
And suddenly found than
/server-status enabled with public access… WTF

1
2
3
4
5
6
7
8
/etc/apache2/mods-enabled # cat ./status.conf

<Location /server-status>
    SetHandler server-status
    Order deny,allow
    Deny from all
    Allow from 127.0.0.1 ::1
</Location>

And part of nginx config

1
2
3
  proxy_set_header   Host             $host;
  proxy_set_header   X-Real-IP        $remote_addr;
  proxy_set_header   X-Forwarded-For  $proxy_add_x_forwarded_for;

Headers looks o.k…

DAMN! I JUST FORGET TO INSTALL RPAF FOR APACHE:(

About apache rpaf
It changes the remote address of the client visible to other Apache modules when two conditions are satisfied. First condition is that the remote client is actually a proxy that is defined in httpd.conf. Secondly if there is an incoming X-Forwarded-For header and the proxy is in it’s list of known proxies it takes the last IP from the incoming X-Forwarded-For header and changes the remote address of the client in the request structure. It also takes the incoming X-Host header and updates the virtualhost settings accordingly. For Apache2 mod_proxy it takes the X-Forwared-Host header and updates the virtualhosts

Nginx purge (invalidate) cache

How to purge cache record in ngix cache via http request.
Today we install awesome nginx plugin in production https://github.com/FRiCKLE/ngx_cache_purge/
How to use it – real life example:

1. Our cache location and proxy settings:

1
proxy_cache_path /var/cache/nginx/proxy_cache_quick levels=1:2 keys_zone=quick_cache:300m max_size=2m inactive=7d;

Read more »

How to test cdn delivery speed via curl

Our company use our own CDN based on nginx caching. 7 high loaded (40 000 RPS per server) servers in 2 datacenters.
And periodically I observer some deviations in delivery time. from 0.15 to 7.5 or even 30 seconds.
We have nginx SLA module + Graphics and monitoring. But I need to test all servers for anomaly delivery time.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
#!/bin/bash

for l in ip1.x.x.x \
         ip2.x.x.x \
         ....
         ipN.x.x.x; do

echo $l;

    for i in {1..1024}; do
    curl -s -w "%{time_total} -- %{time_connect}\n" -o /dev/null --resolve it.randomthemes.com:$l http://it.randomthemes.com/favicon.ico >> ./$l.txt
    done

done

Then analyse ipN.x.x.x.txt any way you like.

cat | sort -n | tail -n 25
etc.

Have a nice day. I really like curl and hope this will help someone.

How to secure wipe file system

Before you cancel rented dedicated server, it`s good practice to secure wipe disc drives. Reboot to recovery console, and:
Use shred, Luke!

1
shred -n 0 -f -v -z /dev/sda

ext4 perfomance tuning

I use following mount options.
In some projects it gives significant performance boost.

errors=remount-ro – need for hardware problem case. Because if disc remains mounted, further writing attempts can deadly damage file system. And one more case – easy monitoring. Just check via zabbix or nagios that you have no ro file system.

noatime, nodiratime – not fix access time. Double check that your applications doesn`t need this.

discard – use trim for SSD drive. In case SATA or SAS this option ignored by system.
commit, nobarrier – dangerous in case of power outage. But for my infrastructure o.k.

ext4 errors=remount-ro,noatime,nodiratime,commit=100,discard,nobarrier

And SED for fstab fixing (I use puppet, chef, fabric).

1
sed -r -i 's/ext4\s+defaults/ext4 errors=remount-ro,noatime,nodiratime,commit=100,discard,nobarrier/' /etc/fstab

nginx error page depends on user browser language

Task – return different pages depends on user browser language.
i.e. different html if backend return error. And for domain it.randomthemes.com always return english error page on backend error.

nginx.conf

1
2
3
4
map $http_accept_language $lang {
    default en;
    ~ru ru;
          }

Server context:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
set $ep /50x.html; #default error page

if ( $host ~* it.randomthemes.com ) {
set $ep /50x.en.html;
}


if ( $lang ~* en ) {
set $ep /50x.en.html;
}

error_page  503          /dinner.html;
error_page  500 502 504  $ep;
error_page  400          /400.html;