iptables kmod auto-loading

Something like this happened to us recently..

http://backstage.soundcloud.com/2012/08/shoot-yourself-in-the-foot-with-iptables-and-kmod-auto-loading/

adjust retention time for carbon and resize whisper files

some graphite storage-schema.conf examples out there only retain data for 1 day.


[default_1min_for_1day]
pattern = .*
retentions = 60s:1d

this comes as a surprise later when you go back to look at your data and its not there...its been rotated out by carbon due to the geometry of the whisper file. the whisper file has a finite size...ya dig?

Update the retention rule in /opt/graphite/conf/storage-schemas.conf


[default_1min_for_1year]
pattern = .*
retentions = 60s:525600

#adjust existing whisper files

find /opt/graphite/storage/whisper -type f -name "*.wsp" | xargs -I{} whisper-resize.py {} 60:5256000

#restart carbon

/usr/bin/python /opt/graphite/bin/carbon-cache.py --config=/opt/graphite/conf/carbon.conf start


root@nagios4.sv3:~$ ls -la /opt/graphite/storage/whisper/prod/web1/apache/
total 43272
drwxr-xr-x 3 root root    4096 Mar 11 20:15 .
drwxr-xr-x 9 root root    4096 Feb 25 05:19 ..
-rw-r--r-- 1 root root 6307228 Mar 11 20:34 busy_workers.wsp
-rwxr-xr-x 1 root root   17308 Mar 11 20:15 busy_workers.wsp.bak

the old file is there, thats how big a file is that keeps data for a single day at 1min interval.

the other file represents a year at 1min interval. makes it easy to do capacity planning for monitoring.

find /opt/graphite/storage/whisper -type f -name "*.wsp.bak" | xargs -I{} rm -f {}

script to get sha-256 hash of myql table descriptions

#!/bin/bash

# Generate SHA-256 hash database schema 
# and a hash of each individual table schema
# then we can see where changes have taken place.

usage() {
cat <

usage: $0 -d DNAME -h DBHOST

  -d  db name
  -h  db host

EOF
exit 1
}

while getopts "d:h:" OPTION; do
  case "$OPTION" in
    d) DB="$OPTARG" ;;
    h) DBHOST="$OPTARG" ;;
    \?) echo "Invalid Option: -$OPTARG" >&2
        usage
        exit 1 ;;
    *) usage
        exit 1 ;;
  esac
done

#enforce argument policy
[[ -z "$DB" ]] && usage;
[[ -z "$DBHOST" ]] && usage;

Q=`echo TRGtZ123Ec234REpKCg== | base64 -i -d -`

echo $DB_VERSION

#dump the schema and hash the whole thing
DBSCHEMA=`mysqldump -h$DBHOST --no-data -p$Q -uroot $DB`
DHASH=`echo $DBSCHEMA | openssl dgst -sha256`
echo "schemadump:"$DHASH

#get tables in the db
TABLES=`mysql --skip-column-names -h$DBHOST -p$Q -uroot $DB -e "SHOW TABLES;"`

#show the tables so we see what it truly is
echo $TABLES;

for i in $TABLES; do 

    TABLESCHEMA=`mysql -h$DBHOST -p$Q -uroot $DB -e "desc $i;"`
    THASH=`echo $TABLESCHEMA | openssl dgst -sha256`
    echo $i:$THASH

done

bash arrays

http://www.thegeekstuff.com/2010/06/bash-array-tutorial/

how to brain transplant a linux system from Dell to HP C-class Blade

How to brain transplant linux:

Use install media to bring new blade host to base OS.
- PXE Boot to rescue mode, follow instructions to shell
- Verify partitions (because these Blades had CentOS installed on them for testing purposes, the partitions should be OK, but best to be sure):

 fdisk -l
 Device    Boot      Start         End    Blocks   Id  System
 /dev/sda1   *           1       6774   54412123+ 83  Linux
 /dev/sda2           6775       7297     4200997+ 82  Linux swap / Solaris

Ensure you've mounted your disk properly with your rescue operation (mount should show /dev/sda1 mounted as /mnt/sysimage/)
 Unmount the rescue proc and sys
 umount /mnt/sysimage/proc
 umount /mnt/sysimage/sys
 umount /mnt/sysimage/dev/pts
 umount /mnt/sysimage/dev
 umount /mnt/sysimage/selinux

Remove the old OS, you don't need that anymore:
 cd /mnt/sysimage/
 rm -rf *
Remake your proc and sys and dev folders:
 mkdir proc sys dev

Take note of the IP you picked up from DHCP on vlan1:
 ifconfig eth0

+++
Login to your source system

Disable crontab schedules for various jobs

Shutdown application services and other running resources on the source system

Cleanup /var/spool/clientmqueue
 find /var/spool/clientmqueue -type f -mtime +1 -exec rm {} \;

Cleanup /home/backups/
 Verify source is not larger than 50GB

Tar > netcat the file system of your source DM over to new device

On your new host:
 nc -l -p 5555 | tar xvvf -

On your source host, in a screen session:
tar cvvf - bin boot etc home lib lib64 lost+found media misc mnt net opt root sbin selinux srv tmp usr var | nc 5555


 *nc on centOS does not have the -q option that modern variants of nc have

Once that is completed (they both should die elegantly) chroot to your new environment, and make appropriate changes to grub, fstab, and mtab, and then run grub-install /dev/sda to install the new MBR to the new drive:

chroot /mnt/sysimage
mount -t proc proc proc
mount -t sysfs sysfs sys
cd dev
MAKEDEV generic
grub-install /dev/sda
cd
vim /etc/mtab (change /dev/sda2 to /dev/sda1)
vim /etc/fstab (change LABEL=/1 to /dev/sda1 and LABEL=SWAP-sda3 to /dev/sda2)
vim /boot/grub/menu.lst (change all hd0,1 to hd0,0)

Shutdown the source system and shut the switchports going to that system.
Remove the mac address line from the network-scripts configs
Reboot the new HP blade server.

Once reconnected to the internet, verify nagios checks are coming back good.
 Deactivate any OMSA specific checks for the DM in Nagios
 Configure the Dell OMSA gear to not startup:
 chkconfig dsm_om_connsvc off
 chkconfig dsm_om_shrsvc off
 chkconfig dsm_sa_ipmi off

Install the HP SIM Software *NOTE: voip1-8.sv3 are i686, and voip9 is cents 6.X*
For Centos 5.X server i686 (voip1-8): wget http://admin1-1.sv3.somedomain.com/hpsim/bootstrap.sh
bootstrap.sh ProLiantSupportPack
For Centos 6.x x64 server (voip9): wget http://admin1-1.sv3.somedomain.com/hpsim/psp-9.10.rhel6.x86_64.en.tar.gz
For Centos 5.x x64 servers (voip10-27): wget http://admin1-1.sv3.somedomain.com/hpsim/psp-9.10.rhel5.x86_64.en.tar.gz

yum install -y hp-health hp-smh-templates hp-snmp-agents hpacucli hpdiags hpmouse hponcfg hpsmh cpqacuxe

cd /tmp
wget http://labs.consol.de/download/shinken-nagios-plugins/check_hpasm-4.6.3.tar.gz
tar zxvf check_hpasm-4.6.3.tar.gz
cd check_hpasm-4.6.3
./configure --enable-hpacucli
make
cp -av plugins-scripts/check_hpasm /usr/local/nagios/libexec/

Added to /usr/local/nagios/etc/nrpe.cfg in command definition section:

command[check_hpasm]=/usr/local/nagios/libexec/check_hpasm $ARG1$

Ran 'visudo' and changed Nagios permitted commands to:

nagios        ALL=(root) NOPASSWD: /usr/sbin/smartctl, /sbin/hpasmcli, /sbin/hpacucli, /usr/sbin/hpacucli
Defaults:nagios !requiretty

uncomment the crontabs

twitter api notes

http://apiwiki.twitter.com/

API is entirely HTTP-based

The Twitter API supports UTF-8 encoding. Please note that angle brackets ("<" and ">") are entity-encoded to prevent Cross-Site Scripting attacks for web-embedded consumers of JSON API output. The resulting encoded entities do count towards the 140 character limit. When requesting XML, the response is UTF-8 encoded. Symbols and characters outside of the standard ASCII range may be translated to HTML entities.

Two APIs - REST and Search.

    The Twitter REST API methods allow developers to access core Twitter data. This includes update timelines, status data, and user information.

    The Search API methods give developers methods to interact with Twitter Search and trends data. The concern for developers given this separation is the effects on rate limiting and output format.


Rate Limiting

    REST API
        150 calls per hour
        The REST API does account- and IP-based rate limiting. Authenticated API calls are charged to the authenticating user's limit while unauthenticated API calls are deducted from the calling IP address' allotment.
        Rate limiting only applies to methods that request information with the HTTP GET command. API methods that use HTTP POST to submit data to Twitter, such as statuses/update do not affect rate limits.
        Can request whitelisting to make up to 20000 requests per hour.

    Search API
        The Search API is rate limited by IP address. The actual limit is not specified but it is quite high.
        requires that applications include a unique and identifying User Agent string. A HTTP Referrer is expected but is not required.

fix drbd split brain

This will force a full sync:

badNode#
drbdadm secondary all
drbdadm disconnect all

(..goes into StandAlone Secondary/Unknown)
drbdadm invalidate all
drbdadm connect all


goodNode#
drbdadm connect all

tunnelblick configuration

SCZ OpenVPN Tunnelblick Configuration File

When  work on the fan-boi hardware, I use http://code.google.com/p/tunnelblick/ with good results.

See below for a simple configuration file I use:

tls-client
dev tap
proto udp
remote 123.456.789.0 443
nobind
pull
persist-key
persist-tun
comp-lzo
verb 3

-----BEGIN CERTIFICATE-----
MIIDsjCCAxugAwIBAgIJAOzhjavkykwEMA0GCSqGSIb3DQEBBAUAMIGYMQswCQYD
VQQGEwJVUzELMAkGA1UECBMCQ0ExEzARBgNVBAcTClNhbnRhIENydXoxJTAjBgNV
BAoTHFJlbGlhbmNlIENvbW11bmljYXRpb25zIEluYy4xFDASBgNVBAMTC3Njb2Zm
aWNldnBuMSowKAYJKoZIhvcNAQkBFhtiaGVuY2tlQHNjaG9vbG1lc3Nlbmdlci5j
b20wHhcNMDgwMTE2MjEwNDQ2WhcNMTgwMSEzMjEwNDQ2WjCBmDELMAkGA1UEBhMC
VVMxCzAJBgNVBAgTAkNBMRMwEQYDVQQHEwpTYW50YSBDcnV6MSUwIwYDVQQKExxS
ZWxpYW5jZS098Db21tdW5pY2F0aW9ucyBJbmMuMRQwEgYDVQQDEwtzY29mZmljZXZw
bjEqMCgGCSqGSIb3DQEJARYbYmhlbmNrZUBzY2hvb2xtZXNzZW5nZXIuY29tMIGf
MA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQC1EHHthfKq6lZMkNht+lpLb0tyGQHY
7quOAtGUEJpzQgWxc6Ebq2pafYy+aenIFi+8B6Z8f1Tisz+pAlc74B9NCoevz44F
L3KiARwXWw5+JTfllinOb9kt9EW07FCejOQ1Uwz/ItUoSoOVLEmrMYDbZMpNkv5y
EuN4BHQzDI54HwIDAQABo4IBADCB/TAdBgNVHQ4EFgQUlpaE91AP8fodDQc2a1Fm
accBcuwwgc0GA1UdIwSBxTCBwoAUlpaE91AP8fodDQc2a1FmaccBcuyhgZ6kgZsw
gZgxCzAJBgNVBsAYTAlVTMQswCQYDVQQIEwJDQTETMBEGA1UEBxMKU2FudGEgQ3J1
ejElMCMGA1UEChMcUmVsaWFuY2UgQ29tbXVuaWNhdGlvbnMgSW5jLjEUMBIGA1UE
AxMLc2NvZmZpY2V2cG4xKjAoBgkqhkiG9w0BCQEWG2JoZW5ja2VAc2Nob29sbWVz
c2VuZ2VyLmNvbYIJAOzhjavkykwEMAwGA1U3EwQFMAMBAf8wDQYJKoZIhvcNAQEE
BQADgYEAiUOk/5DZIOzGPy2oLgME4ih0VfCRndZPkCg6f1bnJ1NUi/tR4GKE3vm5
gkyIrn97AwPwyG+/CHcspbXnZbiw9XqHpnhPugmCud5YQ3QaVgDsGbESmtdeFkHO
JKN9ktl6TrJYsTzc7Y6XwRZ5vCHjofJoOhII8c38gMq6VajQOmA=
-----END CERTIFICATE-----


client cert here


client key here

DKIM New Best Practices In Wake of Disclosed Key Length Vulnerability

http://www.maawg.org/m3aawg-issues-dkim-new-best-practices-wake-disclosed-key-length-vulnerability

ntp.conf

server ntp1 iburst
server ntp2 iburst
driftfile /var/lib/ntp/ntp.drift
statistics loopstats peerstats clockstats
filegen loopstats file loopstats type day enable
filegen peerstats file peerstats type day enable
filegen clockstats file clockstats type day enable
restrict -4 default kod notrap nomodify nopeer noquery
restrict 127.0.0.1
restrict 10.0.0.0 mask 255.0.0.0

editing graphite.db

I'm not all that experienced with Django, or sqlite. But, today I figured out how to delete graph views from the graphite.db to get rid of graphs that cant be deleted through the UI. At first I tried using 'django-admin.py', but the 'manage.py' wrapper was easier to use because I didn't need to create env vars.

#location of manage.py and settings.py
cd /opt/graphite/webapp/graphite
root@monitor1.sv3:/opt/graphite/webapp/graphite$ python manage.py dbshell
SQLite version 3.7.3
Enter ".help" for instructions
Enter SQL statements terminated with a ";"
sqlite>.help

#outputs help here

Most of the time I have trouble deleting graphs from "My Graphs" or "User Graphs", below is an example of an entry where I made a spelling error as well as a poor choice for naming (I used periods instead of underscore, ..oops.).
To do that:

sqlite> select * from account_mygraph;
2|2|Most Deviant Web Servers with respecet to somepoorlynamed.file.php|http://10.80.1.5:8080/render/?width=1564&height=299&_salt=1360994411.086&target=mostDeviant(2%2C%20prod.web*.apache.properlynamed_file_php)&title=Most%20Deviant%20Web%20Servers%20with%20respect%20to%poorlynamed.file.php

Check the schema to get the column name

sqlite> .schema account_mygraph
CREATE TABLE "account_mygraph" (
    "id" integer NOT NULL PRIMARY KEY,
    "profile_id" integer NOT NULL REFERENCES "account_profile" ("id"),
    "name" varchar(64) NOT NULL,
    "url" text NOT NULL
);

#Delete record

sqlite> delete from account_mygraph where id = 2

Done.
Check the UI

This site was helpful: http://www.sqlite.org/sqlite.html

history configuration enhancements

shopt -s histappend
export HISTSIZE=999999
export HISTTIMEFORMAT='%F %T '

dstat commands

dstat -c --top-cpu -d --top-bio --top-latency

dstat -cndymlp -N total -D 5 25

dstat dstat -s --top-io-adv --top-bio-adv

Source: http://dag.wieers.com/home-made/dstat/

rrd's not showing up in graphite

rrd's not showing up in graphite?

On the graphite system, check if the rrd's were written on 64-bit or 32-bit system.

rrdtool info filename.rrd

If the graphite system is a different arch, convert them to xml with:

for i in `ls *.rrd`; do rrdtool dump $i > $i.xml; done

Convert back after moving the files:

for i in `find . -name '*.xml'`; do rrdtool restore $i `echo $i |sed s/.xml//g`; don


This article was helpful.
http://slog.carlheaton.co.uk/index.php/2009/10/converting-32bit-rrd-to-64bit-rrd-moving-cacti-between-architectures/

mongodb main process terminated with status 100

catch 
mongodb does not start
mongodb main process terminated with status 100

try
rm /var/lib/mongodb/mongod.lock

mount /proc on openbsd or freebsd

mkdir /proc
mount -t procfs proc /proc

or

echo "proc /proc procfs rw 0 0" > /etc/fstab 

Nagios Stats 07Feb2013

 Monitoring Performance
Service Check Execution Time: 0.02 / 18.13 / 0.945 sec
Service Check Latency: 0.00 / 0.97 / 0.382 sec
Host Check Execution Time: 0.02 / 0.29 / 0.050 sec
Host Check Latency: 0.00 / 2.40 / 0.304 sec
# Active Host / Service Checks: 201 / 2429
# Passive Host / Service Checks: 184 / 2787

Could not load host key: /etc/ssh/ssh_host_ecdsa_key

on openbsd, u can reload ssh by just running the binary

/usr/sbin/sshd
Could not load host key: /etc/ssh/ssh_host_ecdsa_key

If you see this error, create the ecdsa key with the following command:

ssh-keygen -t ecdsa -f /etc/ssh/ssh_host_ecdsa_key -N ""

disable/enable nagios via command line

curl -k -d "cmd_mod=2&cmd_typ=11" "https://my_nagios_hostname/nagios/cgi-bin/cmd.cgi" -u "your_username:your_password"

 curl -k -d "cmd_mod=2&cmd_typ=12" "https://my_nagios_hostname/nagios/cgi-bin/cmd.cgi" -u "your_username:your_password"

Warning: Unable to get hardware address for interface

First, this started happening:


Starting Nmap 5.21 ( http://nmap.org ) at 2013-02-02 14:18 MST
Warning: Unable to get hardware address for interface re0 -- skipping it.
Warning: Unable to get hardware address for interface re1 -- skipping it.
WARNING: Unable to find appropriate interface for system route to 192.168.1.254
nexthost: failed to determine route to 192.168.1.1
QUITTING!
root:86#

Then I tried this, but it didnt work:

root:86# pkg_add -u http://mirror.servihoo.net/pub/OpenBSD/5.2/packages/i386/libdnet-1.12p4.tgz              
Problem finding http://mirror.servihoo.net/pub/OpenBSD/5.2/packages/i386/libdnet-1.12p4.tgz



But, this resolved both issues.

root:91# export PKG_PATH=http://mirrors.syringanetworks.net/pub/OpenBSD/5.2/packages/i386 

root:92# pkg_add -ui -D update -D updatedepends