Friday, September 30, 2016

Do I need to protect chronyd against dDoS attacks, same like ntpd?


Nope.

For ntp.conf, restrict default kod nomodify notrap nopeer noquery [..] was used. This is not required for chrony. chronyd listens for commands only on the loopback interface and it functions only as an NTP client by default. All NTP requests are ignored. With chrony-2.1.1 the NTP port 123 is not even open and packets send to that port won't reach chronyd.
If chronyd is expected to work also as an NTP server, it is required to add an allow directive to the config.
The kod restrict option in ntp.conf does nothing unless the limited option, which enables rate limiting, is specified too. When kod and limited are used together, ntpd as a server will reply with a KoD packet to clients that are sending too many requests to back off. Unfortunately, clients that do this generally don't understand the KoD reply, so it actually can make things worse and it's better to leave it disabled. chronyd as a server doesn't support rate limiting or KoD. As clients, both ntpd and chronyd support KoD, this is always enabled.
As a general NTP recommendation, it would be good to add a third server to the config, so if one of the servers goes nuts, the other two can outvote it.

from here: https://access.redhat.com/solutions/1977523

Friday, September 9, 2016

dynamic log error discovery using logstash and statsd

logstash configuration using statsd output filter
this took a while to put together, getting the syntax tuned up.
you probably dont need the curly braces on the match statements, but it makes this config backwards compatible with logstash < 1.5

fyr: the routing key will look like:
 "statsd.logstash.`hostname`.appperf.abc.errors.appmodules.ModuleNameFromLog.count"
 "statsd.logstash.`hostname`.appperf.abc.errors.appmodules.ModuleNameFromLog.rate"


## /etc/logstash/conf.d/19-abclog2statsd.conf 

input {
  file {
    path => "/opt/abcapp/tomcat/logs/ABC.log"
  }

}

filter {
  grok {
    match => { "message" => "%{DATE:date} %{TIME:time} ERROR  %{WORD:unknown_module}%{GREEDYDATA:message}" }
    match => { "message" => "%{DATE:date} %{TIME:time} ERROR %{WORD:unknown_module}%{GREEDYDATA:message}" }
  }

}

output {
  if [unknown_module] =~ /.+/ {
    statsd {
      host => "10.101.25.137"
      count => { 
                "appperf.amp.errors.appmodules.%{unknown_module}" => "1"
      }
    }
  }
}



Saturday, February 22, 2014

Internal error:: html instance has no attribute 'multisite_views error in check mk multisite


apt-get install python3 apache2-mpm-prefork

..running omd 1.10 for precise

Monday, December 9, 2013

Ubuntu 12.04 Precise: "libmysqlclient.so.16: no version information available"


This is a difficult issue to troubleshoot on google as there is lots of interference as well as confusion. Here is what I have done to over come this issue, and the issue we're faced with when trying to resolve it with apt-get.

The Error:

/usr/lib/nagios/plugins/check_mysql: libmysqlclient.so.16: no version information available (required by /usr/lib/nagios/plugins/check_mysql)

The Common Sense Solution that doesnt work:

#1 - doesnt work
cd /usr/lib/x86_64-linux-gnu/ && ln -s /usr/lib/x86_64-linux-gnu/libmysqlclient.so.18 libmysqlclient.so.16

#2 - doesnt work
root@pbsiplb1:[/usr/lib/x86_64-linux-gnu]: apt-get install libmysqlclient16
Reading package lists... Done
Building dependency tree
Reading state information... Done
E: Unable to locate package libmysqlclient16


The Solution:

wget http://launchpadlibrarian.net/75954717/libmysqlclient16_5.1.58-1ubuntu1_amd64.deb
dpkg -i ./libmysqlclient16_5.1.58-1ubuntu1_amd64.deb

Friday, November 1, 2013

Revert from Java 7 from Oracle to the Apple-provided Java SE 6 web plug-in and Web Start functionality


http://support.apple.com/kb/HT5559??

This acually worked for me...when are they going to stop putting white spaces in directory names?


sudo mkdir -p /Library/Internet\ Plug-Ins/disabled

sudo mv /Library/Internet\ Plug-Ins/JavaAppletPlugin.plugin /Library/Internet\ Plug-Ins/disabled

sudo ln -sf /System/Library/Java/Support/Deploy.bundle/Contents/Resources/JavaPlugin2_NPAPI.plugin /Library/Internet\ Plug-Ins/JavaAppletPlugin.plugin

sudo ln -sf /System/Library/Frameworks/JavaVM.framework/Commands/javaws /usr/bin/javaws

Sunday, July 14, 2013

Cannot find module (IANAifType-MIB)

you can fix these errors:

Cannot find module (IANAifType-MIB): At line 13 in /usr/share/mibs/ietf/IF-MIB
Did not find 'IANAifType' in module #-1 (/usr/share/mibs/ietf/IF-MIB)
Cannot find module (IANA-RTPROTO-MIB): At line 14 in /usr/share/mibs/ietf/IP-FORWARD-MIB
Did not find 'IANAipRouteProtocol' in module #-1 (/usr/share/mibs/ietf/IP-FORWARD-MIB)
No log handling enabled - turning on stderr logging
MIB search path: /var/www/.snmp/mibs:/usr/share/mibs/site:/usr/share/snmp/mibs:/usr/share/mibs/iana:/usr/share/mibs/ietf:/usr/share/mibs/netsnmp
Cannot find module (IANAifType-MIB): At line 13 in /usr/share/mibs/ietf/IF-MIB
Did not find 'IANAifType' in module #-1 (/usr/share/mibs/ietf/IF-MIB)
Cannot find module (IANA-RTPROTO-MIB): At line 14 in /usr/share/mibs/ietf/IP-FORWARD-MIB
Did not find 'IANAipRouteProtocol' in module #-1 (/usr/share/mibs/ietf/IP-FORWARD-MIB)
No log handling enabled - turning on stderr logging

by running the following:

$ sudo apt-get install snmp-mibs-downloader

but most likely you had those installed, so just do this:

$ sudo download-mibs
$ sudo sed -i 's/^mibs/#mibs/g' /etc/snmp/snmp.conf

Thursday, June 13, 2013

clear cached memory in linux


sync; echo 3 > /proc/sys/vm/drop_caches

Saturday, May 18, 2013

server.gif fix for omd 0.56

cp /opt/omd/versions/0.56/share/nagios/htdocs/images/logos/server.gif /omd/versions/0.56/share/check_mk/web/htdocs/images/icons/

Saturday, April 13, 2013

alias for converting unix epoch time



alias conv="perl -pe 's/(\d+)/localtime($1)/e'"

tail -f nagios.log | conv

Saturday, April 6, 2013

non-root user capture


ssh -X user@host


sudo apt-get install libcanberra-gtk-module libcap2-bin
sudo chgrp wireshark /usr/bin/dumpcap
sudo chmod 750 /usr/bin/dumpcap
sudo setcap cap_net_raw,cap_net_admin=eip /usr/bin/dumpcap
sudo getcap /usr/bin/dumpcap

Launch wireshark
Capture

Wednesday, April 3, 2013

when did linux boot?


jibe02:~# cat /proc/stat
cpu  93878078 0 16855580 25751420315 34959524 105248 583941 0 0
cpu0 17320194 0 4114063 6378518608 8860219 25377 139115 0 0
cpu1 31359989 0 5080216 6433128705 9034045 26296 152290 0 0
cpu2 23660085 0 3881774 6448155438 8538259 27098 140773 0 0
cpu3 21537809 0 3779525 6491617563 8527000 26475 151761 0 0
intr 2015970250 64 0 0 0 0 0 0 0 5 0 0 0 0 0 0 0 0 0 125 0 0 0 0 23 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 7317621 1852240958 156411454 0 0 0 0 0 0 0 0 0
ctxt 49060828436
btime 1301067620
processes 5483876
procs_running 1
procs_blocked 0
jibe02:~# date -R -d @1301067620
Fri, 25 Mar 2011 08:40:20 -0700

Thursday, March 28, 2013

Tests on Linux Real-Time Kernel


http://www.zeromq.org/results:rt-tests-v031

Conclusion: "Our tests prove that the real-time Linux kernel, specifically, SUSE Linux Enterprise Real Time 10 SP2, is capable of eliminating latency spikes. It is expected that our results would be even more favourable for the real-time Linux kernel if these tests were run on boxes loaded with other tasks, rather than on a clean and idle test environment."

Thursday, March 21, 2013

This one reverses the bits in a word


   n = ((n >>  1) & 0x55555555) | ((n <<  1) & 0xaaaaaaaa);
   n = ((n >>  2) & 0x33333333) | ((n <<  2) & 0xcccccccc);
   n = ((n >>  4) & 0x0f0f0f0f) | ((n <<  4) & 0xf0f0f0f0);
   n = ((n >>  8) & 0x00ff00ff) | ((n <<  8) & 0xff00ff00);
   n = ((n >> 16) & 0x0000ffff) | ((n << 16) & 0xffff0000);

Wednesday, March 20, 2013

Graphite chart Y-Axis scale changes with width and height of graph

https://answers.launchpad.net/graphite/+question/152690

I made the chart wider (1400 pixels for 700 minutes of time), thereby having more pixels than horizontal data points.

I was struggling to find the place to manage my legend for each chart, check out the Apply Function menu, and then go down into Special | Add values to legend name | {choices}

Tuesday, March 19, 2013

Great article on NUMA and mysqld

http://blog.jcole.us/2010/09/28/mysql-swap-insanity-and-the-numa-architecture/

Wednesday, March 13, 2013

Nagios OCP Daemon Howto

https://wiki.icinga.org/display/howtos/OCP+Daemon

iptables kmod auto-loading

Something like this happened to us recently..

http://backstage.soundcloud.com/2012/08/shoot-yourself-in-the-foot-with-iptables-and-kmod-auto-loading/

Monday, March 11, 2013

adjust retention time for carbon and resize whisper files

some graphite storage-schema.conf examples out there only retain data for 1 day.


[default_1min_for_1day]
pattern = .*
retentions = 60s:1d


this comes as a surprise later when you go back to look at your data and its not there...its been rotated out by carbon due to the geometry of the whisper file. the whisper file has a finite size...ya dig?

Update the retention rule in /opt/graphite/conf/storage-schemas.conf


[default_1min_for_1year]
pattern = .*
retentions = 60s:525600


#adjust existing whisper files

find /opt/graphite/storage/whisper -type f -name "*.wsp" | xargs -I{} whisper-resize.py {} 60:5256000

#restart carbon

/usr/bin/python /opt/graphite/bin/carbon-cache.py --config=/opt/graphite/conf/carbon.conf start


root@nagios4.sv3:~$ ls -la /opt/graphite/storage/whisper/prod/web1/apache/
total 43272
drwxr-xr-x 3 root root    4096 Mar 11 20:15 .
drwxr-xr-x 9 root root    4096 Feb 25 05:19 ..
-rw-r--r-- 1 root root 6307228 Mar 11 20:34 busy_workers.wsp
-rwxr-xr-x 1 root root   17308 Mar 11 20:15 busy_workers.wsp.bak


the old file is there, thats how big a file is that keeps data for a single day at 1min interval.
the other file represents a year at 1min interval. makes it easy to do capacity planning for monitoring.

find /opt/graphite/storage/whisper -type f -name "*.wsp.bak" | xargs -I{} rm -f {}

Saturday, March 9, 2013

script to get sha-256 hash of myql table descriptions

#!/bin/bash

# Generate SHA-256 hash database schema 
# and a hash of each individual table schema
# then we can see where changes have taken place.

usage() {
cat <

usage: $0 -d DNAME -h DBHOST

  -d  db name
  -h  db host

EOF
exit 1
}

while getopts "d:h:" OPTION; do
  case "$OPTION" in
    d) DB="$OPTARG" ;;
    h) DBHOST="$OPTARG" ;;
    \?) echo "Invalid Option: -$OPTARG" >&2
        usage
        exit 1 ;;
    *) usage
        exit 1 ;;
  esac
done

#enforce argument policy
[[ -z "$DB" ]] && usage;
[[ -z "$DBHOST" ]] && usage;

Q=`echo TRGtZ123Ec234REpKCg== | base64 -i -d -`

echo $DB_VERSION

#dump the schema and hash the whole thing
DBSCHEMA=`mysqldump -h$DBHOST --no-data -p$Q -uroot $DB`
DHASH=`echo $DBSCHEMA | openssl dgst -sha256`
echo "schemadump:"$DHASH

#get tables in the db
TABLES=`mysql --skip-column-names -h$DBHOST -p$Q -uroot $DB -e "SHOW TABLES;"`

#show the tables so we see what it truly is
echo $TABLES;

for i in $TABLES; do 

    TABLESCHEMA=`mysql -h$DBHOST -p$Q -uroot $DB -e "desc $i;"`
    THASH=`echo $TABLESCHEMA | openssl dgst -sha256`
    echo $i:$THASH

done

Friday, March 8, 2013

bash arrays

http://www.thegeekstuff.com/2010/06/bash-array-tutorial/