Delete lines from a file by line number with sed

Today i decided to make a new monitoring tool, and I needed to make a list of all permutations of 3 in a set of 22. The set happens to be hostnames of a private Tor network. Order is important, as forming 3-hop circuits through Tor is sequential, which is why i need permutations instead of combination's.

22 * 21 * 20 = 9240 permutations

Crap, I'm not really up on my combinatorial number theory, I guess I'll have to hack it up.

First I used an excel plugin to generate all the permutations.

But this ended up giving me 1408 invalid permutations, because the mix of sets had 10647 results. I copied the results into a text file and counted the number of lines as well as obtained the line numbers of the invalid permutations using this script:
----------
#!/bin/sh

tornames=("tornode01" "tornode02" "tornode03" "tornode04" "tornode05" \
"tornode06" "tornode07" "tornode08" "tornode09" "tornode10" \
"tornode11" "tornode12" "tornode13" "tornode14" "tornode15" \
"tornode16" "tornode17" "tornode18" "tornode19" "tornode20" \
"tornode21" "tornode22")

for i in ${tornames[*]};
do
while read line; do echo $line|tr " " "\n"|grep $i |wc -l; done < ./vc_list.bak > ./lines.$i
grep -rn '3\|2' ./lines.$i | cut -d: -f1 > ./lines.$i.ln
done
----------

It saved a bunch of files for me as: lines.[hostname], containing a number on each line indicating the number of times the hostname appears on each line.
Then it grep'd out the lines with a 2 or a 3, asking grep to return the line number, and cut the line number from the output to a file named: lines.[hostname].ln

Then at the command line I did this:

# cat ./lines.*.ln > line.numbers.all
# sed 's/.*/&d/g' ./line.numbers.all > ./delete.sed
# sed -f delete.sed ./file.master >> file.trimmed

Using a sed delete file...finally I had my 9240 valid permutations:

# cat ./file.trimmed | wc
9240 27720 254520

Next I want to make this text list into an array that I can `source` into the monitoring script as an array.

# rsync ./file.trimmed ./perms_array.sh
sed -i -e 's/^\./perms_array.sh

Almost done, I just need to fill in the array number with another sed expression.

# sed = ./perms_array.sh | sed 'N; s/^// ; s/\nperms\[// ; s/^/perms\[/' > \
./perms_array.final.sh

and now to put quotes around the array value:

# sed -e 's/\=/\=\"/' < ./perms_array.final > ./perms_array.final.new && rsync ./perms_array.final.tmp ./perms_array.final
# sed -e 's/$/\"/' < ./perms_array.final > ./perms_array.final.new && rsync ./perms_array.final.tmp ./perms_array.final

here's what the file looks like:

perms[1]="tornode01 tornode02 tornode03"
perms[2]="tornode01 tornode04 tornode05"
perms[3]="tornode01 tornode06 tornode07"
...

Now I can move on to write an essentially simple script that performs the test of all possible virtual circuits.

oops

Today I was trying to move a file into the home directory of the current user, like this:

# mv /home/otheruser/somefile ~

Interestingly enough, after doing this for the last 15 years, I fat-fingered it like this:

# mv /home/otheruser/somefile !

CRAP!
Guess what, my file was deleted.
Btw, Mac OSX doesnt behave like this. I assume *BSD, Solaris and other high-quality systems dont as well. Pooor Linux.

SNMPv3 Quickstart

i DONT claim this to be complete or authoritative. But, with these quick steps i was able to get snmpv3 working, a generally avoided version of a widely used protocol, and a fog to many sysadmins i've worked with. I'm sick of reading 10 pages of prose to get the steps i need to move securely forward in my projects.

0) yum install net-snmp.i386 net-snmp-libs.i386

1) Run snmpconf -i to create snmpd.conf and snmp.conf
- sudo /usr/bin/snmpconf -i

Notes:
- if you're regenerating the files at some point, cd into /etc/snmp, then run `snmpconf -i`. snmpconf looks in the local dir for files first before looking elsewhere. The resultant files are still saved to /usr/local/share/snmp/
- when creating snmp.conf, complete section 3: 2-10
- when creating snmpd.conf, complete section 4: 1-3
- when creating a new user while configuring snmpd.conf, choose 'priv' for the minimum security level. you can also restrict the user to a specific branch of the OID tree here as well.

2) Copy these files to /etc/snmp

rsync -av /usr/local/share/snmp/snmp.conf /etc/snmp/
rsync -av /usr/local/share/snmp/snmpd.conf /etc/snmp/

3) Run net-snmp-config to actually create snmpv3 user, here is the correct syntax...

net-snmp-config --create-snmpv3-user [-ro] [-A authpass] [-X privpass] [-a MD5|SHA] [-x DES|AES] [username]

Here's my command that corresponds to my previous configuration of snmp.conf and the test snmpget command further below in step 5:

net-snmp-config --create-snmpv3-user -ro -A authpass -X privpass -a SHA -x AES rouser

Note: the manpage for net-snmp-config has the X and x incorrectly in their example of create-snmpv3-user. The help cruft (for net-snmp-config --help) shows it correctly. I tried to create a read-write user (with -rw), but it didnt work. I dont change system parameters through snmp anyways, so it doesnt matter to me. Maybe the absence of [-ro] creates a read-write user? seems like ro should be the default unless -rw is specified. ???

4) Restart snmpd service

5) Make test snmpv3 request

snmpget -v 3 -n "" -u rouser -a SHA -A "authpass" -x AES -X "privpass" -l authPriv localhost system.sysUpTime.0

Notes:
- The `-l authPriv` argument specifies that the request should be both signed (-a SHA) and encrypted (-x AES).
- The command above can be greatly simplfied because most of these options have been declared in the snmp.conf.

install a perl module

perl -MCPAN -e 'install Net::SNMP'

or

perl -MCPAN -w -e 'shell'

CPAN> install Net::SNMP

Deployment Tools: Puppet

I've started looking at Puppet as the next gen tool for system deployments. Check it out.

http://reductivelabs.com/trac/puppet/wiki/DocumentationStart

It leaves cfengine and others in the dust...

List all Perl Modules

perl -MFile::Find=find -MFile::Spec::Functions -Tlwe "find { wanted => sub { print canonpath $_ if /\.pm\z/ }, no_chdir => 1 }, @INC"

Passmark Health Check

curl https://localhost:443/pmws_server/healthCheck -k

Running VMware ACE Player as a Windows Service

First, read this. I wasnt able to use the resource kit tools because Macrosh@ft wont allow you to redistribute their tools. So, we instead bought a tool that offered an OEM license. If you dont know what ACE is, its vmware's option pack for VMware Workstation. Its basically a bunch of security features and packaging options, meaning, you can build vm and then package it up and install it on another system. I wanted to use it because I wanted to make sure the server could not be copied and run somewhere else by anyone except the people I choose and authorize.

One of the features in an ACE policy is that allows you to run a script or executable instead of enter a password when the vm is started. The VMware ACE Player, interestingly enough, will accept a string from STDOUT of this script or exe in order to attempt the decryption of the encrytion key thats used to read the vmdk files as the vm runs. Using FireDaemon and the FireDaemon features of Pre-Post Service commands, I was able to use vmrun.exe stop command to shut down the vm when the host system is rebooted. I also made lanmanworkstation a service that the Firedaemon service depends on. This way, the vm is not started until the network is full up on the host system...other people have mentioned using this technique and it seemed like a good one.

The executeable, written in VC++ STL, accepts two args, a meaningful 9 digit number and a secret. If either of these args fail a number of tests performed on them, the exe quietly exits. If the args pass, a SHA512 hash (using openssl) is performed and a 64 character string is printed out to STDOUT. Its this string that the vmplayer.exe uses to encrypt/decrypt the AES keys that encrypt/decrypt the vmdk's when the vm starts.

These files, the exe and the openssl libs (dll's) and the VC manifest and their dll's and any other script and stuff you want to use in the Firedaemon service configuration, reside in the "ACE Resources" directory, under the parent directory of the vm Master.

Also, i got snagged by this: each time a package is built, the packager drops an ace.sig file into the "ACE Resources" directory of the package it generates. Make sure this ace.sig file doesnt get copied back into the Master "ACE Resources" directory. If it does, all the packages that you make will have this invalid file in there. Its easy to get into this situation when you delete the files in that directory in order to make a test package that may be updated with a policy update package, which would contain the authentication module and scripts. VMware should fix this by checking for this file and deleting it from within the Master's dir structure each time the packager runs, but it doesn't at this time. Anyhow, there is a policy option called Resource Signing that does check this file, and if its set to check it, which it is by default, the activation of the package (or policy update) will fail.

There was one more terribly annoying thing with setting up the service. When I attempted to shutdown or reboot the host system (unforced), the vmplayer.exe would abort the shutdown/reboot operation and display a modal dialog box that says "virtual machine is in use." I got around this by using AutoIT, a freeware application that allows you to create a simple script to operate windows and applications. So far its been a breeze and a very sensible and intuitive scripting language. Good docs help a lot too. Anyhow, this script ran as a Firedaemon pre-service and sits there waiting for that stupid dialog box to activate. It checks for this window every 250ms. Then, when the shutdown sequence starts, it gets rid of the dialog box and reboots the host system. Pretty slick.

Its too bad vmware says they wont support ACE vm's anytime soon on ESX or VMware Server. It would be pretty simple to get this working and I think that vm volumes that remain encrypted on-disk at all times solves a very difficult security challenge in operating system virtualization. no longer can someone make a copy of your entire filesystem and mount it somewhere else.

How to view pflog

Viewing the pflog file:
# tcpdump -n -e -ttt -r /var/log/pflog

A real-time display of logged packets:
# tcpdump -n -e -ttt -i pflog0

 

Install a package manager in MacOSX (Darwin Ports) and install GnuPG port

First, get the package manager:

$ wget http://www.portcode.com/darwinports/DarwinPorts-1.5.0-10.4.dmg
or
$ curl http://www.portcode.com/darwinports/DarwinPorts-1.5.0-10.4.dmg -O

Mount dmg image and install, then open a Terminal window:

$ sudo port -d selfupdate
$ cd /opt/local/var/macports/

$ port search gnupg
$ sudo port install gnupg
Password:
$ gpg --gen-key

conf file is: ~/.gnupg*, add keyservers, change your default fingerprint, and other stuff there.

Deep Water Swell for Northern California

http://cdip.ucsd.edu/?nav=recent⊂=nowcast&units=metric&tz=UTC&pub=public&xitem=sf

Geologic Overlays of Northern California for Google Earth

http://geomaps.wr.usgs.gov/sfgeo/geologic/downloads.html

Squid Fact Sheet

http://www.tonmo.com/science/public/giantsquidfacts.php

Change runlevel Settings for a Group of Services

The following command changes all services set to run at runlevel 5 and sets them to off.

chkconfig --list | grep 5:on | awk '{ print $1 }' | \
 while read LINE; do chkconfig --level 12345 $LINE off; done

Network Access Control

Enterasys has a mature and unmatched product offering, today.

http://www.enterasys.com/products/index.aspx

 

Fact Sheet NSA Suite B Cryptography [08dec2008]

Key length recommendations from the NSA for companies making products for Gov't use.

http://www.keylength.com/en/6/

 

Apache Worker vs. Prefork

This is a good post.

http://www.camelrichard.org/apache-prefork-vs-worker

Good Hints:
Worker is superior in 2+ cpu applications
Compile PHP5 after installing Apache

Restoring Hard Links to Protected Files in Linux

I got this syslog message today:

Dec 4 9:47:56 [hostname] restorecond: Will not restore a file with more than one hard link (/etc/resolv.conf) Invalid argument

Here is how I resolved it:

$ sudo ls -i /etc/resolv.conf # find innode
[inode number] /etc/resolv.conf

$ sudo find /etc -inum [inode number] # find hard links
$ /usr/sbin/lsof | grep resolv.conf # check if file is open
$ mv /etc/sysconfig/networking/profiles/default/resolv.conf ~ # move
$ sudo restorecon /etc/resolv.conf # set selinux defaults
$ sudo ln /etc/resolv.conf /etc/sysconfig/networking/profiles/default/resolv.conf #recreate hard link

 

DDOS Incident Cheat Sheet

This is very well put together:

http://www.zeltser.com/network-os-security/ddos-incident-cheat-sheet.pdf

 

Using SSH Remote and Local Tunnels

The following example demonstrates how to use Remote and Local SSH tunneling to access a service on your highly secure home computer (that's not exposed explicitly through a firewall at your home) from a remote location.

For example:

On Home Computer, before traveling:
ssh -R 44444:localhost:22 user@remote.computer.ip.address

On Remote Computer, after arriving at work:
ssh -L 22222:localhost:44444 user@remote.computer.ip.address (possibly rfc1918 ip)
ssh user@localhost -p22222

Wa la, you now have a SSH session to your home computer from work.