Oooopppps,
Oh shit, seems I really am god after all. It's a good job my grandmother had the good decency to appear to get me out of trouble and hand me L37,000, a good telling off, and an new job. Still when it takes three years to engineer three minutes in the same room and all you personally manage is "Got root (i real your email)" it says a lot about how your family really has some odd issues to sort out. The embarasment nearly killed us both - but the panto was wonderful to behold. Somehow I need to find the time to have five more kids (discovering I already had four was a bit of a shock, especially as one of them is alive and well, but also apparently very much dead).
Oh, I should mention gran was a teacher, so a good telling off was done according to my rules. I dragged my feet learning to read though. Had to pretend to be a Doctor so she would think I was in charge. My teeth hurt becuse of that one. Still, never could lie to gran. I wonder how she'll feel when she discovers what I really mean by 'read' her email. She better not give me dyslexia again.
Welcome to my world everybody.
Loss - feel what I felt. Give me an e and I'll take a t.
If you're wondering what gran looks like, well at the moment - for me - he's hiding in the cat and pretending to shit on my floor. But if you imagine Paris Hilton you'll be in the right sort of ball park.
So if you could stop taking the piss now we'd both appreciate it, thanks.
CARP - Common Address Redundancy Protocol
This was so simple. In fact I imagine the lack of what I would consider proper documentation is precicely because it's so easy.
First I needed to recompile the kernel on both my test boxes. The kernel config is simple enough:
# # /usr/src/sys/i386/conf/CARP # include GENERIC device carp
Then the standard kernel build:
cd /usr/src make -j4 buildkernel KERNCONF=CARP make -j4 installkernel KERNCONF=CARP reboot
Setting-up carp manually was simply a case of copying the config out of the man page. Permanent configuration via /etc/rc.onf wasn't much harder. First on the Host A, the preferred master:
cloned_interfaces="carp0" ifconfig_carp0="vhid 1 pass p4ssword 192.168.0.112"
Then on Host B:
cloned_interfaces="carp0" ifconfig_carp0="vhid 1 advskew 100 pass p4ssword 192.168.0.112"
And that's it. Ping the IP, reboot at random, and it just works.
Generational Backups
A while back I was asked for a script which would ftp a tarball to a server and provide support for multiple generations. Here's my first attempt:
HOST='192.168.254.254' USER='us4rn4m3' PASSWD='p4ssw0rd' ftp -n $HOST <<END_SCRIPT quote USER $USER quote PASS $PASSWD binary delete backup4.tar rename backup3.tar backup4.tar rename backup2.tar backup3.tar rename backup1.tar backup2.tar rename backup0.tar backup1.tar put /usr/tmp/backup.tar backup0.tar quit END_SCRIPT exit 0
Did the job, but it's rudimentary to say the least.
It wasn't long before I was asked to alter the number of generations, and there were other issues. So I wrote a better script, which I've posted in the hope some may find it useful
DNS from a MySQL DB
I've done a lot of DNS in my time. Experience has taught me to be very wary of BIND with MySQL extensions. In fact experience has taught me that it's a really bad idea. So, using my Perl skills I worked out an alternative.
It's very simple. With a simple schema all the zonedata can be thrown into in MySQL. Then a Perl script run from cron generates the zonefiles and configuration, and uses ndc (or rndc) to control the daemon. The rest is standard BIND.
I know this...
Binding a new IP address to an interface. I know this. I do. Really. From the command line it's:
ifconfig fxp0 inet 192.168.144.120 netmask 255.255.255.255 alias
Then, in /etc/rc.conf I just add:
ifconfig_fxp0_alias0="inet 192.168.144.120 netmask 255.255.255.225"
Really it's so simple. But embarrassingly there has been the odd time I've mucked up. So I always like to check.
SSH - and only SSH
I have a box I admin. It's got a public address and a private address. This box does all sorts of interesting things on its private network. Whereas on the internet it just needs to allow me SSH access. I have, of course, secured all the services to prevent then from being exploited via all the evil script kiddies out there. But I want ubersecure. I want the box to accept SSH on the public interface and nothing else - except, of course, if it originates on the box. I've built stateful firewalls in the past, but I've never done so with IPFW - which, for no readilly apparent reason, I fancy doing.
First thing to do is to recompile the kernel with IPFW. I could kldload it, only the I need to change the default rule (I've embarased myself that way once too often). The following is the KERNCONF I used:
include GENERIC options IPFIREWALL options IPFIREWALL_VERBOSE options IPFIREWALL_VERBOSE_LIMIT=100 options IPFIREWALL_DEFAULT_TO_ACCEPT options IPFIREWALL_FORWARD options DUMMYNET options HZ="1000"
Now (see my earlier blog for the specifics) I just need to add the rules:
add 100 allow tcp from any to 10.0.0.1 22 add 110 allow tcp from 10.0.0.1 22 to any add 200 check-state add 300 allow tcp from 10.0.0.1 to any setup keep-state add 310 allow udp from 10.0.0.1 to any keep-state add 900 deny tcp from any to any add 910 deny udp from any to any
All done.
Setup Replication
I find MySQL replication a really handy feature. Only it's painful. The MySQL documentation has cluefull instructions but for some reason I keep forgetting the relevant details and loosing my notes. So as I have two new servers and need for replicated MySQL I may as well place my notes here. Both boxes have mysql-server-4.0.23a freshly installed from the ports.
Master Server
Out of the box MySQL has no config file. This needs to be corrected. Thankfully there are several example configs, one of which is perfect for my needs. So:
cp /usr/local/share/mysql/my-medium.cnf /etc/my.cnf
Now I need a user that is to be used for the replication:
GRANT REPLICATION SLAVE ON *.* TO repl@"%" IDENTIFIED BY 'xxxxxxxx';
There is a way to copy the DBs without stopping MySQL, but as there is nothing using the DB at the moment I'm going to stop it, tar the DBs, and restart it.
/usr/local/etc/rc.d/mysql-server.sh stop tar cvzf mysql-db.tgz -C / var/db/mysql/ /usr/local/etc/rc.d/mysql-server.sh start
Within MySQL's shell I need determine the current binary log name and offset on the master
mysql> show master status; +---------------+----------+--------------+------------------+ | File | Position | Binlog_do_db | Binlog_ignore_db | +---------------+----------+--------------+------------------+ | alpha-bin.003 | 79 | | | +---------------+----------+--------------+------------------+
Copying my tarball to the second server means I'm done with the master server for now.
Slave Server
Again I can copy the example config:
cp /usr/local/share/mysql/my-medium.cnf /etc/my.cnf
But this time I need to change the line:
server-id = 1
to
server-id = 2
Loading the DBs is easy enough - just need to do some tidying:
/usr/local/etc/rc.d/mysql-server.sh stop rm -rf /var/db/mysql tar xvzf mysql-db.tgz -C / find /var/db/mysql/ -type f -depth 1 | xargs rm /usr/local/etc/rc.d/mysql-server.sh start
Now I'm ready to activate the slave:
CHANGE MASTER TO MASTER_HOST='alpha', MASTER_USER='repl', MASTER_PASSWORD='xxxxxxxx', MASTER_LOG_FILE='alpha-bin.003', MASTER_LOG_POS=79; START SLAVE;
Replication Check
Checking replication is working is easy enough to do manually. But I'd like to know when it's not working, and forget about it the rest of the time. So I've written a script which will check the replication for me. There are several ways I could have done this. But inserting an entry into the Master then checking it's replicated to the slave seems like the most confidence inspiring method.
Using the test database I add a table:
USE test; CREATE TABLE _test_( timestamp int( 11 ) NOT NULL default '0', PRIMARY KEY ( timestamp ) );
Then add a MySQL user with the fewest rights I can:
GRANT USAGE ON * . * TO "checker"@"%" IDENTIFIED BY "some_password" GRANT SELECT, INSERT, DELETE ON test._test_ TO "checker"@"%"; FLUSH PRIVILEGES;
And finally I place my script in a suitable location and edit /etc/crontab to run it every 15 mins:
*/15 * * * * xaphod /opt/bin/mysqlrepchk 2>/dev/null
Done!
Ruby on Rails - cookbook
According to http://www.rubyonrails.org/:
Rails is a full-stack, open-source web framework in Ruby for writing real-world applications with joy and less code than most frameworks spend doing XML sit-ups
So far every framework I've ever encounted has sucked. But, of late, it's impossible to lurk on EFnet#freebsduk without somebody mentioning Ruby on Rails (or RoR) in a good light. So I've decived to bite the bullet and take Rails for a spin around the block.
Looking at my system the packages ruby18 and ruby-gems are already installed. And so is a very old version of the rails gem. No idea how that got there... but it takes only a few moments of not knowing what I'm doing to accidentally delete everything and to get the latest and greatest rails installed.
The article everyone seems to be using as light intro to RoR talks about Windows (with inherent backslash - not slash - wrongness), MySQLFront, expects Rails 0.9.4, and uses the builtin webserver. Well simply for the hell of it I'm going to use 0.9.5 (and FreeBSD, phpMyAdmin and Apache) and I'll wing it.
First Steps (and initial Apache config)
First this is to create the empty application:
rails ~/var/rails/cookbook
A quick test - by firing up the built-in WEBrick web server:
cd ~/public_html/rails/cookbook
ruby script/server --binding {actual ip}
then pointing my browser to:
http://mybox:3000/
shows it's all working.
Setup of Apache is a breeze. In httpd.conf I just add:
Alias /rails/cookbook /home/xaphod/var/rails/public
<Directory /home/xaphod/var/rails/cookbook/public>
Options ExecCGI FollowSymLinks
AddHandler cgi-script .cgi
AllowOverride all
Order allow,deny
Allow from all
</Directory>
A quick test shows it's working
MyTest (and mod_rewrite pain)
Once I get it through my thick skull - again - that the ONLamp article is using '\', not '/' it's simple to generate a new controller class.
ruby script/generate controller MyTest
However this fails miserably when I try it with my browser. Rails assumes I'm going to be running a unique vhost. I could. But I don't want to. A bit of mod_rewrite head-banging and I discover that in cookbook/public/.htaccess I just need to change this:
RewriteBase /dispatch.cgi
to this:
RewriteBase /rails/cookbook/dispatch.cgi
Editing my new controller is easy enough... once I find the correct file.
MySQL
The rest of the article is easy enough to follow. With the expection of the insistance on using MySQLFront. And people who casualy ignore the MySQL permissions model deserve all the pain they heap upon themselves. So unlike the ONLamp article I'll create MySQL user specifically to access my cookbook DB.
CREATE DATABASE `cookbook` ; GRANT USAGE ON *.* TO cookbook@localhost IDENTIFIED BY "********"; GRANT ALL PRIVILEGES ON cookbook.* TO cookbook@localhost WITH GRANT OPTION ;
And I'll create my DB the easy way:
CREATE TABLE recipes ( id int(10) unsigned NOT NULL auto_increment, title varchar(255) NOT NULL default '', description varchar(255) NOT NULL default '', date date NOT NULL default '0000-00-00', instructions text NOT NULL, category_id int(10) unsigned NOT NULL default '0', PRIMARY KEY (id), UNIQUE KEY title (title) ) TYPE=MyISAM AUTO_INCREMENT=1; CREATE TABLE categories ( id int(10) unsigned NOT NULL auto_increment, name varchar(50) NOT NULL default '', PRIMARY KEY (id) ) TYPE=MyISAM AUTO_INCREMENT=1;
Once I'm done the rest is, as I said, easy enough.
Secure NFS
My usual answer to the question, "How do I secure an NFS server?" is to not run NFS. Really. NFS stands for No Fucking Security. Which is why I'll only ever install NFS servers on totally closed private networks. But recently I had no choice - I had to run NFS and the box had to have a public IP.
I installed 5.3-RELEASE the same way as I usually do, with the exception that I allowed sysinstall to enable the NFS Server. This added the following to /etc/rc.conf:
nfs_server_enable="YES" rpcbind_enable="YES"
A look at sockstat showed the following NFS related listening going on:
root nfsd 377 3 tcp4 *:2049 *:* root nfsd 377 4 tcp6 *:2049 *:* root mountd 375 4 udp4 *:1006 *:* root mountd 375 5 tcp4 *:888 *:* root mountd 375 6 udp6 *:1005 *:* root mountd 375 7 tcp6 *:887 *:* root rpcbind 308 4 udp6 *:* *:* root rpcbind 308 5 stream /var/run/rpcbind.sock root rpcbind 308 6 udp6 *:111 *:* root rpcbind 308 7 udp6 *:1023 *:* root rpcbind 308 8 tcp6 *:111 *:* root rpcbind 308 9 udp4 *:111 *:* root rpcbind 308 10 udp4 *:862 *:* root rpcbind 308 11 tcp4 *:111 *:*
Rather horrific really. But it's not all bad. A quick look in /etc/defaults/rc.conf and a read of the corresponding man pages and I discover that I can bind nfsd and rpcbind to a single IP, and mountd can be told to bind to a specific port and to use libwrap. So now the /etc/rc.conf entries look like this:
nfs_server_enable="YES" nfs_server_flags="-u -t -n 4 -h 172.16.0.1" rpcbind_enable="YES" rpcbind_flags="-h 172.16.0.1 -l" mountd_enable="YES" mountd_flags="-r -p 888"
Thankfully all the hosts that will be accessing this box are on the same subnet - hence the private IP alias. A sockstat now (after a reboot) tells me:
root nfsd 368 3 tcp4 172.16.0.1:2049 *:* root mountd 360 4 udp4 *:888 *:* root mountd 360 5 tcp4 *:888 *:* root mountd 360 6 udp6 *:888 *:* root mountd 360 7 tcp6 *:888 *:* root rpcbind 308 4 udp6 *:* *:* root rpcbind 308 5 stream /var/run/rpcbind.sock root rpcbind 308 6 udp6 ::1:111 *:* root rpcbind 308 7 udp6 *:* *:* root rpcbind 308 8 udp6 *:1023 *:* root rpcbind 308 9 tcp6 *:111 *:* root rpcbind 308 10 udp4 127.0.0.1:111 *:* root rpcbind 308 11 udp4 172.16.0.1:111 *:* root rpcbind 308 12 udp4 *:862 *:* root rpcbind 308 13 tcp4 *:111 *:*
Mildly better, but rpcbind is still talking to the outside world. A fact shown by running rpcinfo from a remote host:
# rpcinfo -p nfsbox
program vers proto port service
100000 4 tcp 111 rpcbind
100000 3 tcp 111 rpcbind
100000 2 tcp 111 rpcbind
100000 4 udp 111 rpcbind
100000 3 udp 111 rpcbind
100000 2 udp 111 rpcbind
100000 4 local 111 rpcbind
100000 3 local 111 rpcbind
100000 2 local 111 rpcbind
100005 1 udp 888 mountd
100005 3 udp 888 mountd
100005 1 tcp 888 mountd
100005 3 tcp 888 mountd
100003 2 udp 2049 nfs
100003 3 udp 2049 nfs
100003 2 tcp 2049 nfs
100003 3 tcp 2049 nfs
A quick change to /etc/hosts.allow like so:
# Secure the port mapper rpcbind : 172.16.0.0/255.255.255.0 : allow rpcbind : ALL : deny
Shows libwrap is doing it's job:
# rpcinfo -p nfsbox rpcinfo: can't contact portmapper: RPC: Authentication error; why = Client credential too weak
Now, having done all I can to make NFS inherently secure, the next step is to throw a firewall around the whole thing. Easily done. Just add this to /etc/rc.conf:
firewall_enable="YES" firewall_script="/etc/rc.firewall" firewall_type="/etc/ipfw.rules" firewall_quiet="NO" #change to YES once happy with rules firewall_logging_enable="NO"
Create /etc/ipfw.rules:
add 010 allow ip from 172.16.0.0/24 to any add 100 set 10 unreach 3 udp from any to any 111 add 110 set 10 unreach 3 udp from any to any 888 add 120 set 10 unreach 3 udp from any to any 2049 add 210 set 10 reset tcp from any to any 111 add 220 set 10 reset tcp from any to any 888 add 230 set 10 reset tcp from any to any 2049 add 65534 allow ip from any to any
And start ipfw:
/etc/rc.d/ipfw start
A word of caution about /etc/ipfw.rules, one error anywhere in the file and no rules are loaded. As the default rule is to deny everything this can lead to embarrassment. Especially when you do it twice and the box in question resides in a data centre to which you don't have access...
With NFS secure all that remains is to edit /etc/exports export the filesystems I need:
/export -alldirs -maproot=0 172.16.0.2
Getting mountd to recognise the canges is simple too:
killall -HUP mountd
Job done - once I've tested the client can mount it of course.
CNAMEs are NOT Evil!
It seems to be another universally accepted truth that CNAMEs are evil. It also seems to be something else I've discovered I disagree with. Having just been saved a mass of effort thanks to CNAMEs I have to say CNAMEs are nothing but a handy tool available to DNS administrators.
My task is to migrate a webserver from a box in one data-centre to another box in a different data-centre. Speed is of the essence as there is a network outage planned over the weekend. Transferring data is not hard, a copy of rsync and some time is all that's needed. Configuring the new Apache is easy enough too, even if the migration is from Apache/1.3.23 to Apache/2.0.50. But with 654 virtual servers configured, updating DNS is likely to be a right pain.
Just to get these relevant Resource Records changed looks tricky. Of the 654 domains which correspond to the virtual servers, 5 are lame, 171 no longer exist, 356 are on a nameserver I administer, the remainder are spread out over 45 different nameservers. As each one of these is a different customer of a customer that's 122 major headaches... I'm also a conservative techie. So I like to plan for the worst. Hence I'd really like to be able to change things back quickly should things go wrong. So the first thing to do is to set the Time To Live on the current RR to 1 hour, then... Bah, the whole thing suddenly seems too horrible to contemplate.
However my investigations revealed that mostly the relevant RRs have been setup with CNAMEs, and I have control of the canonical name. So suddenly I have just 7 headaches which I'll happily throw back to Mr Customer.
Handy they may be, but CNAMEs should not be used lightly. The scope to break things is huge. In a previous job seeing a CNAME and OTHER data error was far too common - and virtually impossible to explain how to avoid them to non-combatants. So even though I don't think CNAMEs are evil, I think I'll happily keep telling people they are...
Building Apache with Static Modules
I recently had need to trim back an Apache config to disable all the fluff. Specifically I disabled all modules which were not required for the servers normal running, and commented out the related config. For what I need these are the onlt modules I require:
LoadModule access_module libexec/apache2/mod_access.so LoadModule log_config_module libexec/apache2/mod_log_config.so LoadModule env_module libexec/apache2/mod_env.so LoadModule setenvif_module libexec/apache2/mod_setenvif.so LoadModule mime_module libexec/apache2/mod_mime.so LoadModule autoindex_module libexec/apache2/mod_autoindex.so LoadModule dir_module libexec/apache2/mod_dir.so LoadModule alias_module libexec/apache2/mod_alias.so
However, the ultimate Apache optimisation is to statically link modules. Which, last time I investigated, could not be done via the ports. It must have been a long time ago that I did my noodling. Because I notice the other day the ports do now support it. Time to reinvestigate.
cd /usr/ports/www/apache2
The first thing I discovered is that make now has three additional options:
show-options show-modules show-categories
Reading the make show-options suggets this should do the trick:
make WITH_STATIC_MODULES="access log_config env setenvif mime autoindex dir alias" install clean
However, this does not work as I intended. The modules I want static are static. But so are the ones I want dynamic:
/usr/local/sbin/httpd -l Compiled in modules: core.c mod_access.c mod_auth.c mod_include.c mod_log_config.c mod_env.c mod_setenvif.c prefork.c http_core.c mod_mime.c mod_status.c mod_autoindex.c mod_asis.c mod_cgi.c mod_negotiation.c mod_dir.c mod_imap.c mod_actions.c mod_userdir.c mod_alias.c mod_so.c
Ah, well. Never mind.
Chroot SSH
For the longest time I've had a problem with ftp. My problem is, specifically, that I dont like enabling ftp on any server for which I'm responsible. Like telnet, I feel it's fundamentally insecure. It's also an absolute pain in the arse trying to get ftp to play nicely with firewalls.
Obviously SSH is the way to go. I've use scp to copy my files around for years, more recently I discovered the joys of sftp when trying to get OpenSSH to talk to SSH2. The problem is, however, that SSH gives you a shell. So it's not a drop-in replacement for ftp. Furthermore, it's seemingly impossible to drop users into a chrooted environment.
For a while I looked at VShell from VanDyke Software. But as it's commercial it's a non-starter (I insist on open source). But a perusal through the ports when I found myself with an hour to kill and I discovered something which seeminigly met my requirements.
Installing scponly was simple enough:
cd /usr/ports/shells/scponly/ make -DWITH_SCPONLY_CHROOT -DWITH_SCPONLY_RSYNC install clean
Setting-up a chrooted user is also simple enough too:
cd /usr/local/share/examples/scponly/ ./setup_chroot.sh
Example output is below. Only one caveat: the script will barf with a rather unhelpful message if it can't find config.h in the same directory as the script is run from.
Next we need to set the home directory for this scponly user. please note that the user's home directory MUST NOT be writeable by the scponly user. this is important so that the scponly user cannot subvert the .ssh configuration parameters. for this reason, a writeable subdirectory will be created that the scponly user can write into. -en Username to install [scponly] -en home directory you wish to set for this user [/home/scponly] -en name of the writeable subdirectory [incoming] creating /home/scponly/incoming directory for uploading files Your platform (FreeBSD) does not have a platform specific setup script. This install script will attempt a best guess. If you perform customizations, please consider sending me your changes. Look to the templates in build_extras/arch. - joe at sublimation dot org please set the password for scponly: Changing local password for scponly. New password: Retype new password: passwd: updating the database... passwd: done if you experience a warning with winscp regarding groups, please install the provided hacked out fake groups program into your chroot, like so: cp groups /home/scponly/bin/groups
A quick test with scp and sftp, and finally I've acheived something that's been on my TO DO list for far too long. Not ideal, as I hate needing all the shrubbery. But it'll do.
rsync
For the second time in as many days I've got to setup rsync to run as a daemon. Only yesterday I never bothered to make notes - which is a pain. I think I better remedy that situation...
First install rsync:
cd /usr/ports/net/rsync make install clean
Now configure the SOURCE box:
#
# /usr/local/etc/rsyncd.conf
#
pid file = /var/run/rsyncd.pid
use chroot = yes
[share]
path = /export/share
comment = Share directory
auth users = user
secrets file = /usr/local/etc/rsyncd.secrets
And:
# # /usr/local/etc/rsyncd.secrets # user:clearpassword
Make the rsyncd.secrets private:
chmod 0600 /usr/local/etc/rsyncd.secrets
And start:
echo "rsyncd_enable=\"YES\"" >> /etc/rc.conf /usr/local/etc/rc.d/rsyncd.sh start
On the DESTINATION this does the business:
rsync -av --delete rsync://user@123.123.123.123/share /export/share
As this will prompt fot the password it's not suitable for cron. However:
echo "clearpassword" > /usr/local/etc/rsyncd.passwd.share chown cronuser /usr/local/etc/rsyncd.passwd.share chmod 0600 usr/local/etc/rsyncd.passwd.share
Now add this to the correct crontab:
/usr/local/bin/rsync -a --delete --password-file=/usr/local/etc/rsyncd.passwd.share rsync://user@123.123.123.123/share /export/share 2>&1 >/dev/null
And relax in the knowlege that your data is safe..
Exim Log
Being a techie, reading the manual is something of a rarity. The Exim Specification, however, is one thing I do read. Just occasionally I find the need to make some notes. Being on the terse side, Exim's logging is occasionally hard to fathom. Having discovered my notes on this subject are somewhat out of date I've updated them...
Log Flags
| <= | message arrival |
| => | normal message delivery |
| -> | additional address in same delivery |
| *> | delivery suppressed by -N |
| ** | delivery failed; address bounced |
| == | delivery deferred; temporary problem |
Fields
| A | authenticator name (and optional id) |
| C | SMTP confirmation on delivery |
| CV | certificate verification status |
| DN | distinguished name from peer certificate |
| DT | time taken for a delivery |
| F | sender address (on delivery lines) |
| H | host name and IP address |
| I | local interface used |
| id | message id for incoming message |
| P | on <= lines: protocol used |
| on => lines: return path | |
| QT | time spent on queue |
| R | on <= lines: reference for local bounce |
| on => lines: router name | |
| S | size of message |
| ST | shadow transport name |
| T | on <= lines: message subject (topic) |
| on => lines: transport name | |
| U | local user or RFC 1413 identity |
| X | TLS cipher suite |
Message Reception
2002-10-31 08:57:53 16ZCW1-0005MB-00 <= kryten@dwarf.fict.example
H=mailer.fict.example [192.168.123.123]:9767
I=[82.148.225.15]:25 U=exim
P=smtp S=5678 id=<incoming message id>
X=TLSv1:DES-CBC3-SHA:168 CV=no
T="Rimmer Directive #271"
- 2002-10-31 08:57:53
- date and time
- 16ZCW1-0005MB-00
- Message ID
- kryten@dwarf.fict.example
- envelope sender address
- A bounce message is shown with the sender address "<>"
- if locally generated 'R=<message id>' is a reference to the message that caused bounce it
- H=mailer.fict.example [192.168.123.123]:9767
- host name, IP address, port
- Ommitted for locally generated messages
- The number given in square brackets is the IP address of the sending host.
- If there is a single, unparenthesized host name in the H field it has been verified to correspond to the IP address (see the host_lookup option).
- If the name is in parentheses, it was the name quoted by the remote host in the SMTP HELO or EHLO command, and has not been verified.
- If verification yields a different name to that given for HELO or EHLO, the verified name appears first, followed by the HELO or EHLO name in parentheses.
- Only the final address in square brackets can be relied on.
- I=[82.148.225.15]:25
- incoming_interface & incoming_port
- U=exim
- local user or RFC 1413 identity (ident lookups not implemented)
- P=smtp
- protocol for incoming message
- X=TLSv1:DES-CBC3-SHA:168
- TLS cipher suite
- S=5678
- Size
- X=TLSv1:DES-CBC3-SHA:168
- TLS cipher suite
- S=5678
- size of message
- id=<incoming message id>
- incoming message id
- T="Rimmer Directive #271"
- message subject (topic)
- "No chance you metal bastard"
Message Delivery
There are 2 formats. First is for remote deliveries:
2002-10-31 08:59:13 16ZCW1-0005MB-00 => marv <marv@hitch.fict.example>
R=localuser T=remote_smtp S=1234
H=mailer.fict.example [192.168.123.123]
C="250 2.0.0 i1G0Bjif055100 Message accepted for delivery"
QT=6s DT='0's
The second is for local deliveries:
2002-10-31 09:00:10 16ZCW1-0005MB-00 => monk@holistic.fict.example
R=dnslookup T=local_delivery S=1234
QT=10s DT=4s
For ordinary local deliveries, the original address is given in angle brackets after the final delivery address, which might be a pipe or a file. If intermediate address(es) exists between the original and the final address, the last of these is given in parentheses a fter the final address. The R and T fields record the router and transport that were used to process the address.
- generation of a reply message by a filter file gets logged as a "delivery" to the addressee, preceded by ">"
- second and subsequent addresses are flagged with "->" instead of "=>"
- When two or more messages are delivered down a single SMTP connection, an asterisk follows the IP address in the "[-=]>" lines
Greppage
Using grep(1) to pull info from logfiles is useful. But it is mildly annoying as multiple log lines refer to individual mails. The exigrep utility is a Perl script that searches one or more main log files for entries that match a given pattern. When it finds a match, it extracts all the log entries for the relevant message, not just those that match the pattern. Thus, exigrep can extract complete log entries for a given message, or all mail for a given user, or for a given host, for example.
The usage is:
exigrep [-l] [-t<n>] <pattern> [<log file>] ...
The -t argument specifies a number of seconds. It adds an additional condition for message selection. Messages that are complete are shown only if they spent more than <n> seconds on the queue.
The -l flag means "literal", that is, treat all characters in the pattern as standing for themselves. Otherwise the pattern must be a Perl regular expression. The pattern match is case-insensitive. If no file names are given on the command line, the standard input is read.
IPFW
Way back in the midsts of time I picked IPF as my FreeBSD firewall of preference. It was a simple choice to make. There were two firewalls on offer. IPFW didn't handle state. In FreeBSD 4.0 IPFW became stateful. But there never seemed much reason top change my allegiance. Except for DUMMYNET. Which I just knew I'd want to play with one day. Well, that day arrived just the other day, when I found I needed to limit the traffic from a webserver.
Working out the correct way to configure IPFW didn't appear to be documented anywhere. Not that I really looked hard. When ipfw(8) didn't tell me precisely what I wanted I trawled the rc files manually. In /etc/defaults/rc.d I found this:
firewall_enable="NO" # Set to YES to enable firewall functionality firewall_script="/etc/rc.firewall" # Which script to run to set up the firewall firewall_type="UNKNOWN" # Firewall type (see /etc/rc.firewall) firewall_quiet="NO" # Set to YES to suppress rule display firewall_logging="NO" # Set to YES to enable events logging firewall_flags="" # Flags passed to ipfw when type is a file
Which pointed me to /etc/rc.firewall and the following:
# Define the firewall type in /etc/rc.conf. Valid values are: # open - will allow anyone in # client - will try to protect just this machine # simple - will try to protect a whole network # closed - totally disables IP services except via lo0 interface # UNKNOWN - disables the loading of firewall rules. # filename - will load the rules in the given filename (full path required)
Control freak that I am I wanted my own setup. So I created /etc/ipfw.rules Whacked in a pipe, and a rule to divert traffic through it:
pipe 1 config bw 128KBytes/s mask src-ip 123.123.123.123 add 10 set 1 pipe 1 tcp from 123.123.123.123 80 to any out via fxp0
Then it was just a simple case of amending /etc/rc.conf like so:
## Firewall stuff firewall_enable="YES" firewall_script="/etc/rc.firewall" firewall_type="/etc/ipfw.rules" firewall_quiet="NO" #change to YES once happy with rules firewall_logging_enable="NO"
Then running the rc script /etc/rc.d/ipfw start to fire it up.
Only thing. The script /etc/rc.firewall only flushes rules, not pipes. So a small edit was needed just in case I ever decide to hack the rules and restart the firewall... because I just know I'll never remember to flush the rules manually. Really simple:
--- rc.firewall-orig Thu Jun 5 02:57:21 2003
+++ rc.firewall Sun Oct 3 21:15:49 2004
@@ -106,6 +106,7 @@
# Flush out the list before we begin.
#
${fwcmd} -f flush
+${fwcmd} -f pipe flush
############
# Network Address Translation. All packets are passed to natd(8)
Job done.
Binaries in CVS
CVS rocks. Except when it comes to binary files. I'm always forgetting how to handle them properly. So:
>cvs add -kb foo.png cvs add: scheduling file 'foo.png' for addition cvs add: use 'cvs commit' to add this file permanently
And for them times I forget to remember:
>cvs admin -kb foo.png RCS file: /cvsroot/img/foo.png,v done >cvs update -A foo.png
Keyboard Hint
You would not believe the number of time I've had to reboot a headless FreeBSD server merely to get it to see the keyboard. Yet no matter how many times I've done it I've always forgotten to investigate a permanent fix.
Until now.
cd /boot cp device.hints device.hints.old cat device.hints | sed 's/atkbd.0.flags="0x1"/atkbd.0.flags="0x0"/' > device.hints
Only issue is that the keyboard lights don't. But I can live with that.
So easy. Is going to save me so much time. Now why the hell didn't I investigate sooner?
How strange.
It seems Wiggy has finally decided that Apple's are worth a look. Honestly, it's true. He's apologising for his hypocritical behaviour over on IRC as I write this. Thing is, I'm of the opinion his anti-Apple rhetoric was little more dogmatic rantings. So it's nice to see he's willing to reevaluate with an open mind.
Now, I've always wanted an Apple.
Only the other day I had the misfortune to be in PC World. Overcoming the urge to scream I wandered around the PCs looking at the specifications, laughing at the mistakes, and wondering which bloody price applied to which sodding computer. The urge to kick the next brat to bump into me was almost overwhelming.
When I spied the Apples. Specifically the I-Macs: I've always wanted one you see. So I wandered over to oggle.
Only... well... they looked crap. Nothing like the pictures I've seen, whilst being exactly what the pictures had shown me. The hemispherical base looked comically large. And the shiny white plastic looked dirty and cheap. The swivel arm was cool, only it didn't give me the fine positioning I'd have expected, it sort of jumped about a bit. Not forgetting the speakers which looked tiny, cheap, with the air of something obviously annoying on any desk. The amount of ingrained dirt in the cones made me cringe.
I just knew in my house: with the cat and kids: an I-Mac would look equally crap within a week. Which is why I walked quickly away.
Answers on a postcard.
Take one webserver running Apache/1.3.27, with PHP/4.3.0, on 4.5-RELEASE. Aside from a small problem with it maxing out 4Mbps there is nothing wrong. Now, migrate to Apache/2.0.49, with PHP/4.3.7, on 5.2.1-RELEASE-p9. As the DNS is changed what the load on the new server creep sky high and, eventually, stop responding.
Try again with Apache/1.3.31 on 5.2.1-RELEASE-p9. Then with Apache/2.0.49 on 4.10-RELEASE. Each time what the load average go through the roof: up to 420 at one point: totally locked up. Run top and systat -vmstat 1 and see the box struggle - each of the numerous apache process running with at least 3% CPU.
Turn off softupdates (battery backed RAID controller so we should be fine). Install an Intel fxp NIC and enable interrupt mitigation. At last an improvement... the box grinds to a halt slightly slower.
In desperation copy the binaries off the old 4.5-RELEASE server. Perform some jiggery-pokery to get the dependencies sorted. And fire it up. Then watch top and systat and see as everything runs smoothly and the new server happily sucks up 8Mbps. Marvel at the reduced number of apache processes which only use 0.00% CPU. Wipe the sweat off your brow and mutter "thank f**k for that".
So. What the hell is going on?
The only thing I can possibly think of is Apache's DSO support. Looking at the configure command from the old:
./configure \ "--with-layout=Apache" \ "--prefix=/usr/local/apache-1.3.27" \ "--enable-module=so" \ "--enable-module=all" \ "--enable-shared=expires" \ "--enable-shared=headers" \ "--enable-shared=rewrite" \ "--enable-shared=speling" \ "--enable-shared=unique_id" \ "$@"
And comparing it with the new:
CC="cc" \
CFLAGS="-O -pipe -mcpu=pentiumpro \
-DDOCUMENT_LOCATION=\\\\\"/usr/local/www/data\\\\\" \
-DDEFAULT_PATH=\\\\\"/bin:/usr/bin:/usr/local/bin\\\\\" \
-DHARD_SERVER_LIMIT=512" \
LDFLAGS="-L/usr/local/lib" \
LD_SHLIB="cc" \
INCLUDES="-I/usr/local/include" \
./configure \
"--prefix=/usr/local" \
"--server-uid=www" \
"--server-gid=www" \
"--with-perl=/usr/local/bin/perl" \
"--with-layout=FreeBSD" \
"--datadir=/usr/local/www" \
"--htdocsdir=/usr/local/www/data" \
"--cgidir=/usr/local/www/cgi-bin" \
"--without-confadjust" \
"--enable-module=most" \
"--enable-module=auth_db" \
"--enable-module=mmap_static" \
"--disable-module=auth_dbm" \
"--enable-shared=max" \
"--enable-rule=EXPAT" \
"$@"
However Apache's comments on the matter say:
DSO has the following disadvantages:
- The server is approximately 20% slower at startup time because of the symbol resolving overhead the Unix loader now has to do.
- The server is approximately 5% slower at execution time under some platforms because position independent code (PIC) sometimes needs complicated assembler tricks for relative addressing which are not necessarily as fast as absolute addressing.
Which seems to suggest that DSOs are not going to kill your server. The fact I've never had a problem with DSOs before - not even on moderately loaded servers - simply reinforces this assumption. Besides, PHP is loaded as a DSO on all the varieties of Apache I've been playing with. But on a heavily loaded server could the DSO mechanism slow things enough to cause the system to thrash about?
Sysinstall & Disklabel defaults.
My first FreeBSD install was 2.2.5_RELEASE onto an Olivetti 486SX2 with 8MB of RAM. The Olivetti has been sitting unused in my attic for years, whereas I seem to have lost my original 2.2.5_RELEASE Cd's. Both these fact sadden me somewhat. But I still remember the fun I had. Owing to my overwhelming need to actually understand what's going on I must have reinstalled it a dozen times. Looking stuff up - Alta-vista, I believe, being the search engine of choice at the time - when I failed to understand. Having mainly worked with DOS/Windows and Novell operating systems there was a lot I really couldn't understand.
One of the first things which gave me a problem was the setup of the disks. I still get confused with the partitioning/slicing/labelling nomenclature. But at least I have a fundamental understanding of why it all seems to fly in the face of excepted convention. And more importantly I know the why the different filesystems are different filesystems. Although it seems that many people don't. Which is why I'm always having to deal with machines which fly in the face of best-practise. Logs in non-standard locations (with not a symlink in sight) - because /var is too small - is my personal favourite.
The fault lies, in my opinion, with the Auto Defaults option in sysinstall's Disklabel Editor. In almost all circumstances what this gives you is wrong. Consider the following:
Filesystem 1M-blocks Used Avail Capacity Mounted on /dev/da0s1a 247 54 173 24% / /dev/da0s1d 247 0 227 0% /var /dev/da0s1e 247 0 227 0% /tmp /dev/da0s1f 32148 833 28743 3% /usr
This is the df output from a fully configured running server for which I'm responsible. And here is another one:
Filesystem 1M-blocks Used Avail Capacity Mounted on /dev/amrd0s1a 247 54 173 24% / /dev/amrd0s1d 247 0 227 0% /var /dev/amrd0s1e 247 0 227 0% /tmp /dev/amrd0s1f 65095 922 58965 2% /usr
At this point I'm someone will tell me that I shouldn't have used the auto-defaults. Well, I didn't. Somebody else setup these boxes, by rote, doing it the same way he's always setup a FreeBSD box. Next someone will be telling me to educate this person. Well, it just so happens that this particular person does not care to be educated - besides "they would have made it so that the defaults would be suitable for most things" is the usual response - so the battle is lost before it's begin.
My opinion is that there should be a change. I'm perfectly aware that having an opinion on this matter is going to cause a bikeshed and copious amounts of abuse. But hey-ho... As I see it there are two options.
- Change the default size of /var and /tmp
- By default make a single large filesystem
Although, as yet, I have no I idea which I'd prefer. But I'd definitely prefer a change.