Friday, January 04, 2008

FTP Using a Shell Script

Manually transferring a file or files via FTP is a common and convenient method for moving data from one computer to another, especially if it’s a non-recurring event. But the reality is there are many times when an event is recurring and calls for immediate automation. This can be done by using a simple UNIX script file, which can then be executed via command line interface or added to the crontab. In the shell script, myftp.sh, example below, I'm FTP’ing binary type files (pictures) from a local computer to a remote while logging the activity.

Note: Some organizational policies may not allow login/password information in a script file.

# vi myftp.sh
#! /bin/sh
REMOTE='esoft'
USER='anyuser'
PASSWORD='myftp125'
FTPLOG='/tmp/ftplog'
date >> $FTPLOG

ftp -n $REMOTE <<_FTP>>$FTPLOG
quote USER $USER
quote PASS $PASSWORD
bin
cd /myraid/dailyjpgs
mput *.jpg
quit
_FTP
:wq!

Run via CLI
# ./myftp.sh

Add it to the crontab
# crontab -e


Alternate post: FTP Using One-Liner and Perl Script

Per commenter's request: (call login and password outside of script)

# cat > /export/ftp/.user.txt
anyuser
control+d

# cat > /export/ftp/.passwd.txt
myftp125
control+d

# vi myftp.sh
#! /bin/sh
REMOTE='esoft'
USER=`cat /export/ftp/.user.txt`
PASSWORD=`cat /export/ftp/.passwd.txt`
FTPLOG='/tmp/ftplog'
date >> $FTPLOG

ftp -n $REMOTE <<_FTP>>$FTPLOG
quote USER $USER
quote PASS $PASSWORD
bin cd /myraid/dailyjpgs
mput *.jpg
quit
_FTP
:wq!

20 comments:

Bobby said...

I would add '-i" to ftp command arguments as well. ie: ftp -in

"-i Turns off interactive prompting during multiple file transfers."

esofthub said...

Thank you for the comment bobby.

I tried the -in flags and the only item populated in /tmp/ftplog was the date. Yes, the files transferred without prompting me for a confirmation.

Here's the output
# more /tmp/ftplog
Sat Jan 5 02:57:27 KST 2008

---

I reconfirmed the -n flag and the /tmp/ftplog was populated with date and filenames. The files successfully transferred without prompting me for a confirmation even though there are "?'s" in the logfile.

Here's its output
# more /tmp/ftplog
Sat Jan 5 02:57:27 KST 2008
Sat Jan 5 03:02:14 KST 2008
mput 1.jpg? mput 10.jpg? mput 11.jpg12.jpg? mput 13.jpg? mput 14.jpg? mput 15.jp
g? mput 2.jpg? mput 20.jpg? mput 3.jpg? mput 4.jpg? mput 5.jpg? mput 6.jpg? mput
7.jpg? mput 8.jpg? mput 9.jpg?

Noah said...

I usually prefer to use scp with non-privileged accounts on the "sending" end, and a passwordless public key. Generate an SSH key then push it to the remote box:

bash-3.2$ ssh-keygen
Enter passphrase (empty for no passphrase): (blank)
Enter same passphrase again:

bash-3.2$ ssh axon@10.211.55.4 'cat >> .ssh/authorized_keys' < .ssh/id_rsa.pub
axon@10.211.55.4's password: (my password)


After that, ssh (and by proxy, scp) will work without a password from this machine from now on:
bash-3.2$ ssh axon@10.211.55.4 'uname -a'
OpenBSD chimerabsd.labs.buzzsec.com 4.2 GENERIC#375 i386


Then you can make an easy script that you can call with cron or just run off the CLI occasionally:

bash-3.2$ cat scp.sh
#!/bin/sh
REMOTE='10.211.55.4'
USER='axon'
FTPLOG='/tmp/ftplog'
date >> $FTPLOG
cd ~/Photos
scp * $USER@$REMOTE:myphotos/


bash-3.2$ ./scp.sh


One thing to note is that scp doesn't allow you to redirect output, so you may go fancy and add a "for" loop which echoes the filenames out individually after copying. Cheers, and keep up the blog :)

esofthub said...

Noah, good tip for those that can't put the password in the script file.

I'm hoping to keep the blog going in 2008. We'll see. :)

Thanks.

Anonymous said...

great tip noah. Only thing I would add for newbies like myself would be

when creating the ssh-keygen on a RedHat/Fedora system use

ssh-keygen -t rsa

then after you move the key over to the remote system if you still get asked for the password have the sysadmin do a chmod 700 on your .ssh directory & chmod 600 on the authorized_keys file.

Was pulling my hair out on this for about an hour. :)

Anonymous said...

#! /bin/sh
REMOTE='esoft'
USER='anyuser'
PASSWORD='myftp125’
FTPLOG=’/tmp/ftplog’
date >> $FTPLOG

ftp -n $REMOTE <<_FTP>>$FTPLOG
quote USER $USER
quote PASS $PASSWORD
bin
cd /myraid/dailyjpgs
mput *.jpg
quit
_FTP

In the above script if dailyjpgs folder doesn't exist, how to find the folder existing or not?

esofthub said...

"In the above script if dailyjpgs folder doesn't exist, how to find the folder existing or not?"

The script assumes that the "remote directory" is known (present).

Anonymous said...

Is there a solution to this problem without using '-n'..

I'm trying to automate an FTP process but the remote server doesn't allow anonymous login.. any suggestions?

esofthub said...

Anon

Try this site
http://www.robvanderwoude.com/ftp.html

pedro penduko said...

I am using FreeBSD 6.2 and when I use "-in", mput does not work. When I use "-n" only it works but ftplog only contains a date stamp. solution has been to use 'prompt off' and 'verbose on'. Output was

Fri Apr 4 01:46:16 PHT 2008
Interactive mode off.
Verbose mode on.
local: hello.txt remote: hello.txt
229 Entering Extended Passive Mode (|||51785|)
150 Opening ASCII mode data connection for 'hello.txt'.
226 Transfer complete.
7 bytes sent in 00:00 (8.49 KB/s)
local: hello1.txt remote: hello1.txt
229 Entering Extended Passive Mode (|||51786|)
150 Opening ASCII mode data connection for 'hello1.txt'.
226 Transfer complete.
7 bytes sent in 00:00 (8.82 KB/s)
local: hello2.txt remote: hello2.txt
229 Entering Extended Passive Mode (|||51787|)
150 Opening ASCII mode data connection for 'hello2.txt'.
226 Transfer complete.
7 bytes sent in 00:00 (8.87 KB/s)
221 Goodbye.

Will try to modify to where it emails you if the transfer was not successful

esofthub said...

Pedro Penduko,

"Will try to modify to where it emails you if the transfer was not successful"

When you get it done, please post your results.

Thanks for the comment.

kailash said...

in the above script, we are referring the user name and password in the script itself. Is there any way to hide the username and password by calling them in another file i.e. the file should be the input for the username and password... i tried with this but it doesnt work...i think you can help me in this....

esofthub said...

kailash,

"...Is there any way to hide the username and password by calling them in another file i.e. the file should be the input for the username and password..."

I've added to the post to accommodate this query.

Anonymous said...

what we have to put in place of quote in script? i tried putting quote itself it says command not found

esofthub said...

Anon,

quote is a ftp command. It has to be inside the ftp routine. Unix shell doesn't recognize it.

Naga said...

is there any way to satisfy telnet? Analogous script which telnets instead of ftp.

esofthub said...

Naga,

I don't know.

Daniel said...

Thanks for the script, seems to work except for one major point.

It uploads my file (pie.txt for this example) but when I open pie.txt from the FTP server it is empty. It seems to me that it is creating an empty file but not uploading the contents?

How would I go about fixing this?

Alok said...

I am trying to ftp a file from one machine to another (dir structure is same at both the machines).
Below is the code .


#!/bin/ksh
HOST='abc'
USER='zz'
PASSWD='zz2'
fnm=alok

source_dir='/export/file'
target_dir='/export/file'

ftp -n $HOST <<END_SCRIPT
quote USER $USER
quote PASS $PASSWD

bin
prompt
lcd $source_dirpeed
cd $target_dir


put $fnm
bye
END_SCRIPT
exit 0

However i see an error ,while executing the script
=======================

+ftp -n abc
+ << END_SCRIPT^M
: Name or service not known

Om att nå höjderna said...

What does the <<_FTP>> mean?