Download an XML feed with wget

For one of my projects, I was attempting to download a 70MB XML feed. Unfortunately, it was taking forever to download via HTTP and crashing my MSIE 6.0 and Firefox 2.0.0.7 browsers. I ended up using the wget utility to complete the task. Here's what I did.

C:\Documents and Settings\esoft\Desktop>wget www.xxxx.com/partnerfeeds/esofthub%20lists/esofthub%20xmllists.xml
--02:02:09-- http://www.xxxx.com/partnerfeeds/esofthub%20lists/esofthub%20xmllists.xml
=> `esofthub xmllists.xml'
Resolving www.xxxx.com... xx.1xx.x.2xx
Connecting to www.xxxx.com|xx.1xx.x.2xx|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 70,722,387 (67M) [text/xml]

100%[====================================>] 70,722,387 130.78K/s ETA 00:00

02:08:32 (182.75 KB/s) - `esofthub xmllists.xml' saved [70722387/70722387]

If you don't have wget, here's a couple site you can go to get it.

Download WGET for Windows
Download GNU WGET

Ping Multiple Workstations and Mail Results

Here's a simple Z Shell script to check for network connectivity. If you have several dozen workstations, manually pinging each one can be a laborious task. This script pings (w/2 second timeout) and mails the results to a group alias. You can automate by scheduling it with a cron job.

# vi pinger.zsh
#!/bin/zsh

log=/tmp/log.`date '+%d%h%Y%H%M'`

echo >> $log
date >> $log
echo >> $log

for i in {1..30}
do
ping wkstn$i 2
if [ $? = 0 ]
then
echo wkstn$i up >> $log
else
echo wkstn$i down >> $log
fi
done
cat $log | mailx -s "Workstation Status: `basename $log`" groupalias@esofthub.com
#rm $log
:wq!

Use .htaccess to Deny Internet Protocol (IP) Address or Domain Name

You can use the .htaccess file to deny sites that are wreaking havoc on your site. I have a dozen or so IPs from a couple countries that are doing just that to one of my sites. You can block specific IPs/IP blocks or deny by domain names.

The first two lines (deny statements) are indicative of what I did. By the way, I don't control the main configuration of the host (server), so I had to use this file.

# vi .htaccess
order allow,deny
allow from all

deny from xxx.xxx.xx.xxx
deny from xxx.xxx.xxx.xxx
deny from xx.xxx.xx.
deny from xxx.xx.
deny from xx.
:wq!

Here's a screenshot (so tags shows up on this Blogger post) with the tags:









Extra:
If you are trying to block IPs on the Blogger.com platform, see this post: How to block IPs on Blogger.com

Joining and Renaming Multiple Files Using a for loop

I'm doing this personal project where I parsed almost 26,000 items into a particular format. I then parsed 300 of those items into file with a unique filename, so I ended up with 86 unique files. Lastly, I added a header and footer to each unique file and then renamed it to its original filename. Here is a pithy example of what was done on this task.

# sh
# for i in `ls *_num_*`
do
cat header $i footer > $i.$$
mv $i.$$ $i
echo $i done
done

Extra: For those that asked me how to rename files, the mv command is used to rename files in UNIX. In the example below, the esofthub filename will be renamed to esoft
e.g. # mv esofthub esoft

Delete Tabs, Newlines, Carriage Returns, or Spaces from a File

There are times when you might want to delete formatting characters from a file. The tr command does an excellent job of performing this task. Here are some examples.

Note: I have included a screen shot of this task per the request of a commenter. This example in the jpeg deals with tabs and newlines.

Remove tabs
# more filename
esoft esoft esoft
esoft esoft esoft

# cat filename | tr -d space '\t'
esoftesoftesoft
esoftesoftesoft
# tr -d space '\t' < filename

Remove newlines
# cat filename | tr -d space '\n'
esoft esoft esoftesoft esoft esoft










Additional information:
Remove carriage returns
# cat filename | tr -d space '\r'

Remove spaces
# cat filename | tr -d space "space"

Evaluate NFS Statistical Information with nfsstat

The nfsstat command is used to show the statistical information of Network File System (NFS) and Remote Procedure Call (RPC). It is useful for showing NFS activity between the server and client. My box is down right now, so I won't be able to show you a run right now. But here are a few quick examples.

Show the nfs stats
# nfsstat

Evaluate NFS mount stats between server and the client
# nfsstat -m

Reinitialize the nfs stats
# nfsstat -z

My SysAd Blog Reader Appreciation Shout

I want to thank all the "My SysAd Blog" readers that Stumble this site now and then. It's really nice to see that on a daily basis. It hasn't been Stumbled or Dugg like some of the Linux blogs out there but that was never the objective. But at any rate, I definitely appreciate the stumbles and thank you for helping me share the information. Maybe I will get lucky one of these days...I guess my content should improve first :)

By the way, this is a terse "howto type" blog, so if you're looking for verbosity, that's another (or see man page) blog, so please checkout my technical blogroll :)

Respectfully,

Roy Wood

Try my favorite Free browser:
Get FireFox

My Favorite Links

Linux Screw

The UNIX Forums

Unix Guru Universe

Mika's Blog

PC Blade Computing

Planet Sysadmin

Aspiring Sysdmin

Operating Systems - Unix, Linux, Mac OS X, Windows

TechEd Bloggers

PatchLog | patches 'n hacks

@unixville

The Sys Admin

Tech@Sakana

Unix Admin Corner

Geeky Bits

Bash Shell Cures Cancer

Ramblings of a Geek

Open Source Fedora Core

Schneier on Security

Setup a nisplus Server and Populate its Tables

Setup a nisplus server and then populate its tables. Here's a run on my box. Some of the non-interactive portion of the run has been omitted.

# cd /etc
# cp -p /etc/nsswitch.conf /etc/nsswitch.conf.sav
# cp -p /etc/nsswitch.nisplus /etc/nsswitch.conf
# domainname esofthub.com | tee /etc/defaultdomain
# domainname
esofthub.com

For safey reasons, use copies of the /etc files. Copy those /etc files to a staging area /var/nisfiles.

# cd /var
# mkdir nisfiles; chmod 755 nisfiles
# cd /etc
# cp -p hosts rpc services netmasks networks netgroup services protocols /var/nisfiles
# cd /var/nisfiles

Create empty files. It's recommended that you don't populate the contents of the local passwd and shadow files across the namespace.

# touch passwd shadow ipnodes timezone bootparams group aliases auto_home auto_master ethers

Bourne Shell
# PATH=$PATH:/usr/lib/nis; export PATH

C Shell
# setenv PATH ${PATH}:/usr/lib/nis
# csh
# setenv PATH ${PATH}:/usr/lib/nis
# nisserver -v -r -d esofthub.com

This script sets up this machine "esoft" as an NIS+
root master server for domain esofthub.com..

Domain name : esofthub.com.
NIS+ group : admin.esofthub.com.
NIS (YP) compatibility : OFF
Security level : 2=DES

Is this information correct? (type 'y' to accept, 'n' to change) y

This script will set up your machine as a root master server for
domain esofthub.com. without NIS compatibility at security level 2.
Use "nisclient -r" to restore your current network service environment.
Do you want to continue? (type 'y' to continue, 'n' to exit this script) y
setting up domain information "esofthub.com." ...
setting up switch information ...
switch configuration file already set to use NIS+.
killing process keyserv ...
restarting process keyserv ...
killing NIS and NIS+ processes ...
killing process ypbind ...
killing process rpc.nisd ...
killing process rpc.nispasswdd ...
killing process nis_cachemgr ...
stopping nscd ...

setup NIS_GROUP environment variable ...

rm /var/nis files ...
running nisinit ...
This machine is in the "esofthub.com." NIS+ domain.
Setting up root server ...

All done.

starting root server at security level 0 to create credentials...
running nissetup to create standard directories and tables ...
org_dir.esofthub.com. created
groups_dir.esofthub.com. created
passwd.org_dir.esofthub.com. created
group.org_dir.esofthub.com. created
auto_master.org_dir.esofthub.com. created
auto_home.org_dir.esofthub.com. created
bootparams.org_dir.esofthub.com. created
cred.org_dir.esofthub.com. created
ethers.org_dir.esofthub.com. created
hosts.org_dir.esofthub.com. created
ipnodes.org_dir.esofthub.com. created
mail_aliases.org_dir.esofthub.com. created
sendmailvars.org_dir.esofthub.com. created
netmasks.org_dir.esofthub.com. created
netgroup.org_dir.esofthub.com. created
networks.org_dir.esofthub.com. created
protocols.org_dir.esofthub.com. created
rpc.org_dir.esofthub.com. created
services.org_dir.esofthub.com. created
timezone.org_dir.esofthub.com. created
client_info.org_dir.esofthub.com. created
auth_attr.org_dir.esofthub.com. created
exec_attr.org_dir.esofthub.com. created
prof_attr.org_dir.esofthub.com. created
user_attr.org_dir.esofthub.com. created
audit_user.org_dir.esofthub.com. created

adding credential for esoft.esofthub.com...

Enter login password:
nisaddcred: WARNING: password differs from login password.
Retype password:

creating NIS+ administration group: admin.esofthub.com. ...
adding principal esoft.esofthub.com. to admin.esofthub.com. ...
updating the keys for directories ...
restarting NIS+ root master server at security level 2 ...
killing process rpc.nisd ...
restarting process rpc.nisd ...
starting NIS+ password daemon ...
starting NIS+ cache manager ...
modifying the /etc/init.d/rpc file ...
starting Name Service Cache Daemon nscd ...

This system is now configured as a root server for domain esofthub.com.

You can now populate the standard NIS+ tables by using the
nispopulate script or /usr/lib/nis/nisaddent command.

# nispopulate -v -F -p /var/nisfiles -d esofthub.com

NIS+ domain name : esofthub.com.
Directory Path : /var/nisfiles

Is this information correct? (type 'y' to accept, 'n' to change) y

This script will populate the standard NIS+ tables for domain
esofthub.com. from the files in /var/nisfiles:
auto_master auto_home ethers group hosts ipnodes networks passwd protocols services rpc netmasks bootparams netgroup aliases timezone auth_attr exec_attr prof_attr user_attr audit_user shadow
**WARNING: Interrupting this script after choosing to continue
may leave the tables only partially populated. This script does
not do any automatic recovery or cleanup.
Do you want to continue? (type 'y' to continue, 'n' to exit this script) y
auto_master.org_dir.esofthub.com. OK...
populating auto_master table from file /var/nisfiles/auto_master...
adding standard key-value table auto_master...
adding /var/nisfiles/auto_master to table auto_master.org_dir.esofthub.com.

0 entries added/updated
...
...
Credentials have been added for the entries in the
hosts and ipnodes and passwd table(s). Each entry was given a default
network password (also known as a Secure-RPC password).

This password is:
nisplus

Use this password when the nisclient script requests the
network password.
...
...
Check to see if nisplus was setup
# nisls
esofthub.com.:
org_dir
groups_dir
# niscat host.org_dir
localhost localhost 127.0.0.1
localhost loghost 127.0.0.1
esoft esoft 192.168.1.19
esoft mailhost 192.168.1.19

To remove NISPLUS
Visit this link: remove NIS+ from server

Forward Mail to a Different Account with .forward

There are times when you might need to forward your mail to another email account. The .forward file is used to accomplish this task. Here are a couple examples.

All email will be forwarded to this email address, esoft@esofthubInTheWoods.com.

# echo "esoft@esofthubInTheWoods.com" > $HOME/.forward
Or
# cat > $HOME/.forward
esoft@esofthubInTheWoods.com
control ^d

Output Strings in Lowercase -- UNIX

I should have done this post earlier when I was talking about capitalizing strings. I just forgot. Anyways, here's an easy way to output strings as lowercase.

# cat uppercasefile | tr "[:upper:]" "[:lower:]"
this file's contents are in uppercase
but its output should be
lowercased

# tr "[:upper:]" "[:lower:]" < uppercasefile
this file's contents are in uppercase
but its output should be
lowercased

# cat uppercase | tr '[A-Z]' '[a-z]'
this file's contents are in uppercase
but its output should be
lowercased

# tr '[A-Z]' '[a-z]' < uppercasefile
this file's contents are in uppercase
but its output should be
lowercased

Output lowercase strings to file
# tr '[A-Z]' '[a-z]' < uppercasefile > outputfile_lower

# date | tr "[:upper:]" "[:lower:]"
sat sep 8 23:24:20 kst 2007

Traverse an Argument List using shift Command

You can move through an argument list using the shift command. Here's an example using an inline while loop and and an inline for loop.

# sh

The while loop
# set M Y S Y S A D B L O G
# while [ $# -gt 0 ]
> do
> echo $*
> shift
> done
M Y S Y S A D B L O G
Y S Y S A D B L O G
S Y S A D B L O G
Y S A D B L O G
S A D B L O G
A D B L O G
D B L O G
B L O G
L O G
O G
G

The for loop
# set M Y S Y S A D B L O G
# for i in $*
> do
> echo $*
> shift
> done
M Y S Y S A D B L O G
Y S Y S A D B L O G
S Y S A D B L O G
Y S A D B L O G
S A D B L O G
A D B L O G
D B L O G
B L O G
L O G
O G
G

Print a File in Landscape Mode via Command Line

I was asked how to print a file in landscape mode via the command line. I haven't done that in awhile, so I had to delve into my old notes. Here's the syntax.

# /usr/lib/lp/postscript/postprint -pland filename | lp

Set File or Directory Access Control List with setfacl

The setfacl command is used to add, modify, delete an ACL entry (or entries) or replace the entire Access Control List (ACL) on a file or directory. Here are a few examples.

Replaces the entire ACL via man page

--setfacl -s user:esoft:rwx,user::rwx,group::rw-,mask:r--,other:--- testfile
or its octal equivalent
--setfacl -s user:esoft:7,user::7,group::6,mask:4,other:0 testfile

Before changing ACL
# getfacl testfile
# file: testfile
# owner: root
# group: other
user::rw-
group::r-- #effective:r--
mask:r--
other:r--
# ls -l testfile
-rw-r--r-- 1 root other 0 Sep 4 23:26 testfile

After changing ACL
# setfacl -s user:esoft:rwx,user::rwx,group::rw-,mask:r--,other:--- testfile
# getfacl testfile

# file: testfile
# owner: root
# group: other
user::rwx
user:esoft:rwx #effective:r--
group::rw- #effective:r--
mask:r--
other:---
# ls -l testfile
-rwxr-----+ 1 root other 0 Sep 4 23:27 testfile

Same as above but in octal

# rm testfile
# touch testfile
# ls -l testfile
-rw-r--r-- 1 root other 0 Sep 4 23:30 testfile
# setfacl -s user:esoft:7,user::7,group::6,mask:4,other:0 testfile
# getfacl testfile

# file: testfile
# owner: root
# group: other
user::rwx
user:esoft:rwx #effective:r--
group::rw- #effective:r--
mask:r--
other:---
# ls -l testfile
-rwxr-----+ 1 root other 0 Sep 4 23:32 testfile

#######################################

Change user and mask
# setfacl -m u:esoft:6,m:6 testfile
# getfacl testfile

# file: testfile
# owner: root
# group: other
user::rwx
user:esoft:rw- #effective:rw-
group::rw- #effective:rw-
mask:rw-
other:---

Change group and mask
# setfacl -m g:other:4,m:6 testfile
# getfacl testfile

# file: testfile
# owner: root
# group: other
user::rwx
user:esoft:rw- #effective:rw-
group::rw- #effective:rw-
group:other:r-- #effective:r--
mask:rw-
other:---

Delete ACL
# setfacl -d u:esoft:6 testfile
# getfacl testfile

# file: testfile
# owner: root
# group: other
user::rwx
group::rw- #effective:rw-
group:other:r-- #effective:r--
mask:rw-
other:---

Get ACL of testfile then set (apply) to testfile1
# getfacl testfile | setfacl -f - testfile1

Get the Access Control List on a File or Directory

The getfacl command is used to obtain the filename, file owner, file group owner, and its Access Control List or ACL. It will display the ACL for a regular file or a directory. Here are some examples of its use.

On a file
# getfacl newcron

# file: newcron
# owner: esoft
# group: staff
user::rw-
group::r-- #effective:r--
mask:r--
other:r--

On a directory
# getfacl MyDir

# file: MyDir
# owner: root
# group: other
user::rwx
group::r-x #effective:r-x
mask:r-x
other:r-x

Free Virtual Network Computing Remote Control Software

Virtual Network Computing (VNC) is a client/server software package that allows remote access to graphical desktops. Here's some information about VNC at TightVNC.org. The downloads are free under the GNU General Public License and available on several platforms.