Category Archives: Uncategorized

mdadm replace disk

Copy the partition table to the new disk
Dump layout from sda create on sdb

# sfdisk -d /dev/sda | sfdisk /dev/sdb

Add to raid.

# mdadm --manage /dev/md0 --add /dev/sdb1

Verify
# cat /proc/mdstat

Youtrack setup

~ # cat /home/scripts/yourtrack-docker.sh
#!/bin/bash

docker network create –driver bridge youtrack

version=”2022.1.46592″

docker run -it –restart unless-stopped –name youtrack-server-instance
–network youtrack
-v /opt/youtrack/data:/opt/youtrack/data
-v /opt/youtrack/conf:/opt/youtrack/conf
-v /opt/youtrack/logs:/opt/youtrack/logs
-v /opt/youtrack/backups:/opt/youtrack/backups
-p 127.0.0.1:8080:8080
jetbrains/youtrack:$version

+ nginx letsencrypt frontend

root@s14 ~ # cat /home/scripts/nginx-le-docker.sh
#!/bin/bash
docker run
–restart unless-stopped
–name nginx-letsencrypt
–network youtrack
-v /opt/nginx-le/certs:/etc/letsencrypt
-v /opt/nginx-le/conf/youtrack.randomthemes.ru.conf:/etc/nginx/conf.d/default.conf
-e DOMAIN=youtrack.randomthemes.com
-e EMAIL=ksi.sergey@gmail.com
-p MY_EXTERNAL_IP:80:80
-p MY_EXTERNAL_IP:443:443
-d andreilhicas/nginx-letsencrypt


Fail2Ban + nginx access.log

Today morning nagios reports allert that 2 of our small projects inaccessible. HTTP regexp check failed. They related with Caucasian news media and becouse of Armenia and Azerbaijan war someone start DDOS attack.

So what we have to do:
1. Parse nginx logs by eyes :))
2. Determine attack pattern
3. Configure fail2ban
4. Stay allert!

First pattern


117.68.x.x - - [20/Oct/2020:10:28:00 +0000] "GET //ru/search?search_text=qjxk5ENh5IYc HTTP/1.1" 200 10603 "https://it.randomthemes.com//ru/search?search_text=qjxk5ENh5IYc" "Mozilla/5.0 (Linux; Android 9; FIG-LA1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/77.0.3865.92 Mobile Safari/537.36"

com//ru/search? – standart DDOS attack type. search usually heavy operations for many engines (use sphynx, Luke!).
Second pattern


191.102.x.x - - [20/Oct/2020:06:25:21 +0000] "GET / HTTP/1.1" 500 603 "https://it.randomthemes.com/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.122 Safari/537.36"

Huge amount of traffic from the same user agent.


# cat /etc/fail2ban/filter.d/nginx-it.randomthemes.com.local
[Definition]
failregex = ^<HOST> -.*AppleWebKit\/537.36*.
       ^<HOST> - .*https://it.randomthemes.com//ru/search*.
ignoreregex =

~# cat /etc/fail2ban/jail.local
[nginx-it.randomthemes.com]
enabled = true
port = http,https
filter = nginx-it.randomthemes.com
logpath = /var/log/nginx/access.log
maxretry = 2

Check regexp:


#fail2ban-regex /var/log/nginx/access.log /etc/fail2ban/filter.d/nginx-it.randomthemes.com.local

#service fail2ban reload
#fail2ban-client status
#fail2ban-client status nginx-it.randomthemes.com
Status for the jail: nginx-it.randomthemes.com
|- Filter
| |- Currently failed: 42
| |- Total failed: 13608
| `- File list: /var/log/nginx/access.log
`- Actions
|- Currently banned: 23
|- Total banned: 136
`- Banned IP list: 46.162.x.x

Stay alert 🙂 Caucasian hackers not 1337 🙂 and ddos was boring. 3000+ botnet used. Good qualified developers and operation already lives in US, Russia, Turkey, Europe and have no time to play stupid games. So DDOS over. Fail2Ban is beautiful 🙂 but better to use ipset instead iptables.

How to convert GPT to MBR partition linux.

MBR is old, but not depricated. It`s post just to remember.

gdisk /dev/sdb
x
r
g

x – enter expert mode
r – recovery and transformation options (experts only)
g convert GPT into MBR and exit

and thats all…

parted /dev/sdb
p
.....
.....
Partition Table: msdos

ssh key forwarding small tip just to remember

ssh-add ~/.ssh/id_rsa
vi ~/.ssh/config
Host *
User myusername
StrictHostKeyChecking no
Compression yes
ForwardAgent yes
UseRoaming no

now you can
ssh sd1.randomthemes.com
then sftp sd2.randomthes.com
and no need to add public key from sd1 to sd2

How to make vertica backup

In some cases you don`t need 3 node vertica cluster and Ksafety.
We use vertica as very fast column based database + etl and database size only 50 Gb. So we can easy restore vertica from backup and use etl log processing to get actual data.

simple backup commands.

1. Create config file

/opt/vertica/bin/vbr --setupconfig

2. initialize backup storage

$ /opt/vertica/bin/vbr  --task init --config-file /home/dbadmin/leadada_snapshot.ini
Initializing backup locations.
Backup locations initialized.

3. And finally!!

$ vbr.py --task backup --config-file /home/dbadmin/leadada_snapshot.ini
Starting backup of database leadada.
Participating nodes: v_leadada_node0001.
Snapshotting database.
Snapshot complete.
Approximate bytes to copy: 37754604170 of 37754604170 total.
[==================================================] 100%
Copying backup metadata.
Finalizing backup.
Backup complete!

bash options string example

Just to remember.

#!/bin/bash

usage() { echo "Usage: $0 [-s <45|90>] [-p <string>]" 1>&2; exit 1; }

while getopts ":s:p:" o; do
    case "${o}" in
        s)
            s=${OPTARG}
            ((s == 45 || s == 90)) || usage
            ;;
        p)
            p=${OPTARG}
            ;;
        *)
            usage
            ;;
    esac
done
shift $((OPTIND-1))

if [ -z "${s}" ] || [ -z "${p}" ]; then
    usage
fi

echo "s = ${s}"
echo "p = ${p}"

Is heaper threading enabled linux?

How to check is heaper threading enabled at your server.

use /proc/cpuinfo
Some OPS recommend to check HT cpu flag, but according to manuals HT – for some processors mean HyperTransport for others mean Heaper Threading

Better way to count siblings

The kernel data Linux exposes in /proc/cpuinfo will show each logical cpu with a unique processor number. A logical cpu can be a hyperthreading sibling, a shared core in a dual or quad core, or a separate physical cpu. We must look at the siblings, cpu cores and core id to tell the difference.

If the number of cores = the number of siblings for a given physical processor, then hyperthreading is OFF.

cat /proc/cpuinfo | grep siblings | wc -l; cat /proc/cpuinfo | grep cores | wc -l

How to check if zabbix agent insstalled at all infrastructure hosts

We use LXC containers at PH hosts.
containers named like:
rolename$RoleVersion-$InstanceNumber

For example
at host sXX2
appserver1-1
db1-1
db1-2
zabbix3-1

We need to check, if zabbix agents are installed at all infrastructure. One way – user fabric, Luke, as all 80 level DevOps do, or use ssh as all 40 level DevOps do.
Sorry for a little bit 🙂 “hindi” code 🙂 no review was done 🙂
Read more »

How to install unassigned packages ubuntu/debian

We start chef deployment…
Chef deserve additional Huge article, but let`s start:

  echo "deb http://apt.opscode.com/ `lsb_release -cs`-0.10 main" | sudo tee /etc/apt/sources.list.d/opscode.list
  aptitude update
  aptitude -o Aptitude::CmdLine::Ignore-Trust-Violations=true install opscode-keyring
  aptitude -y -o Aptitude::CmdLine::Ignore-Trust-Violations=true install chef