Tag Archives: nginx

Fail2Ban + nginx access.log

Posted by on October 20, 2020 No comments

Today morning nagios reports allert that 2 of our small projects inaccessible. HTTP regexp check failed. They related with Caucasian news media and becouse of Armenia and Azerbaijan war someone start DDOS attack.

So what we have to do:
1. Parse nginx logs by eyes :))
2. Determine attack pattern
3. Configure fail2ban
4. Stay allert!

First pattern



117.68.x.x - - [20/Oct/2020:10:28:00 +0000] "GET //ru/search?search_text=qjxk5ENh5IYc HTTP/1.1" 200 10603 "https://it.randomthemes.com//ru/search?search_text=qjxk5ENh5IYc" "Mozilla/5.0 (Linux; Android 9; FIG-LA1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/77.0.3865.92 Mobile Safari/537.36"

com//ru/search? – standart DDOS attack type. search usually heavy operations for many engines (use sphynx, Luke!).
Second pattern



191.102.x.x - - [20/Oct/2020:06:25:21 +0000] "GET / HTTP/1.1" 500 603 "https://it.randomthemes.com/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.122 Safari/537.36"

Huge amount of traffic from the same user agent.



# cat /etc/fail2ban/filter.d/nginx-it.randomthemes.com.local

[Definition]

failregex = ^<HOST> -.*AppleWebKit\/537.36*.

       ^<HOST> - .*https://it.randomthemes.com//ru/search*.

ignoreregex =





~# cat /etc/fail2ban/jail.local

[nginx-it.randomthemes.com]

enabled = true

port = http,https

filter = nginx-it.randomthemes.com

logpath = /var/log/nginx/access.log

maxretry = 2

Check regexp:



#fail2ban-regex /var/log/nginx/access.log /etc/fail2ban/filter.d/nginx-it.randomthemes.com.local





#service fail2ban reload

#fail2ban-client status

#fail2ban-client status nginx-it.randomthemes.com

Status for the jail: nginx-it.randomthemes.com

|- Filter

| |- Currently failed: 42

| |- Total failed: 13608

| `- File list: /var/log/nginx/access.log

`- Actions

|- Currently banned: 23

|- Total banned: 136

`- Banned IP list: 46.162.x.x

Stay alert 🙂 Caucasian hackers not 1337 🙂 and ddos was boring. 3000+ botnet used. Good qualified developers and operation already lives in US, Russia, Turkey, Europe and have no time to play stupid games. So DDOS over. Fail2Ban is beautiful 🙂 but better to use ipset instead iptables.

How to build nginx deb with new modules

Posted by on October 24, 2016 No comments

Adding third party module to Nginx.
We need several nginx modules which absent in nginx_full ubuntu package.

redis2
and nginx-sla

#Get nginx-sla code
cd /home/build
mkdir nginx-sla
cd nginx-sla
git clone https://github.com/goldenclone/nginx-sla.git

#get nginx-redis code
cd ../
mkdir nginx-redis2
git clone https://github.com/openresty/redis2-nginx-module.git

apt-get install -y dpkg-dev
mkdir /home/build/nginx-redis
cd /home/build/nginx-redis
apt-get source nginx
apt-get build-dep nginx

Search for file in bgin directory
and edit it
/debian/rules
section full_configure_flags should look like this

full_configure_flags := \
            $(common_configure_flags) \
            --with-http_addition_module \
            --with-http_dav_module \
            --with-http_geoip_module \
            --with-http_gunzip_module \
            --with-http_gzip_static_module \
            --with-http_image_filter_module \
                        --with-http_v2_module \
            --with-http_sub_module \
            --with-http_xslt_module \
            --with-stream \
            --with-stream_ssl_module \
            --with-mail \
            --with-mail_ssl_module \
            --with-threads \
            --add-module=$(MODULESDIR)/nginx-auth-pam \
            --add-module=$(MODULESDIR)/nginx-dav-ext-module \
            --add-module=$(MODULESDIR)/nginx-echo \
            --add-module=$(MODULESDIR)/nginx-upstream-fair \
            --add-module=$(MODULESDIR)/ngx_http_substitutions_filter_module \
                        --add-module=/home/build/nginx-redis2 \
                        --add-module=/home/build/nginx-sla

#increase package version
dch -i
#build package

dpkg-buildpackage -us -uc -b

#put into our repo
dput stable ./nginx_1.10.0-0ubuntu0.16.04.5_amd64.changes

And we have new nginx in out wonderful repo 🙂
p.s. It`s better to change package name, and increase version.

nginx proxy_pass and cache regexp location.

Posted by on July 10, 2015 No comments

nginx cannot proxy_pass at regexp location. I made this workaround.
Works great! Now I can cache any static data provided by backend. 🙂 from any location!

location ~* \.(gif|jpg|png|ico)$ {
      rewrite ^.(gif|jpg|png|ico) /$1 break;
      proxy_pass         http://127.0.0.1:8080;
      proxy_redirect     off;
      proxy_set_header    Host             $host;
      proxy_set_header    X-Real-IP        $remote_addr;

      proxy_set_header   X-Forwarded-For  $proxy_add_x_forwarded_for;
      client_max_body_size       150m;
      client_body_buffer_size    128k;
      proxy_connect_timeout      90;
      proxy_send_timeout         90;
      proxy_read_timeout         90;
      proxy_buffer_size          4k;
      proxy_buffers              4 32k;
      proxy_busy_buffers_size    64k;
      proxy_temp_file_write_size 64k;

      proxy_cache cache_common;
      proxy_cache_key "$host|$request_uri";
      proxy_cache_valid 200 302 301 15m;
      proxy_cache_valid 404         10s;
      proxy_cache_valid any          1m;
        }

How to test cdn delivery speed via curl

Posted by on June 8, 2014 No comments

Our company use our own CDN based on nginx caching. 7 high loaded (40 000 RPS per server) servers in 2 datacenters.
And periodically I observer some deviations in delivery time. from 0.15 to 7.5 or even 30 seconds.
We have nginx SLA module + Graphics and monitoring. But I need to test all servers for anomaly delivery time.

#!/bin/bash

for l in ip1.x.x.x \
         ip2.x.x.x \
         ....
         ipN.x.x.x; do

echo $l;

    for i in {1..1024}; do
    curl -s -w "%{time_total} -- %{time_connect}\n" -o /dev/null --resolve it.randomthemes.com:443:$l https://it.randomthemes.com/favicon.ico >> ./$l.txt
    done

done

Then analyse ipN.x.x.x.txt any way you like.

cat | sort -n | tail -n 25
etc.

Have a nice day. I really like curl and hope this will help someone.

Generate Unique Request ID nginx

Posted by on January 21, 2014 1 comment

Task – Need to add unique ID to each user request. External nginx module such as request ID is very unstable, so I write small perl script to generate UUID and add it to header.
nginx embedded perl is extremely fast, and works very well in high loaded production systems.

Required packages:

aptitude install libossp-uuid-perl

/etc/nginx/nginx.conf

http {
...
perl_require "Data/UUID.pm";
perl_set $uuid 'sub {

  $ug = new Data::UUID;
  $str = $ug->create_str();
  return $str;
               }';
... }

Location config:

    location ~ /data/(.+) {
...
...
            proxy_set_header    X-Request-Id    $uuid;
...
}

Huge ammount 400 at nginx access.log

Posted by on August 24, 2012 2 comments

I observe alot of 400 at nginx access.log.
After investigation found, that modern browsers open 2+ simultanious connections, but some of them frequently not used for data transfer, so nginx close them by timeout with 400 code at access.log

hereis my session from chrome:

root@ads1-1:~# tail -f /var/log/nginx/access.log | grep 77.232.X.X
77.232.X.X – – [24/Aug/2012:22:40:36 +0400] “-” 400 0 “-” “-”
77.232.X.X – – [24/Aug/2012:22:40:37 +0400] “GET /adnet/css/adpreview/20120621.css HTTP/1.1” 200 7644 “http://smi2.ru/data/pop/gen.inner.php?bl=32380” “Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.1 (KHTML, like Gecko) Chrome/21.0.1180.79 Safari/537.1”

So, don`t panic if your found a huge ammount of 400 at access.log