« Coalition does the Opposition’s job | Home | Democracy alive and well in Australia »
How to use ftp in a shell script
By Martin English | November 9, 2005
I often need to FTP a file (or series of files) from one machine to another. Usually, I can do the transfer interactively, but when I have a large file (or series of files) to transfer, it’s convenient to fire it off, disconnect from that Customer’s network.
This requires a little bit of automation that has eluded me in the past, but I finally figured it out.
The Problem
The problem I always encountered in scripting ftp transfers involved getting a password to the ftp server. Typical ftp client programs under the major *NIX variants, Linux, Solaris, and NetBSD all read the ftp password from /dev/tty.
Example (non-working) script
#!/bin/sh
HOST='remote.host'
USER='theirname'
PASSWD='theirpw'
FILE='file.txt'
ftp $HOST < <END_SCRIPT
user $USER
$PASSWD
put $FILE
quit
END_SCRIPT
exit 0
The above script will just hang if run in the foreground (in an terminal window), or if run in the background (from a cron job); it will fail to perform the work of transferring file.txt.
/dev/tty is a strange device. Each process (or more to the point, each process group) has a different one, and you can not naively make ftp clients read the password from some fixed location. Well, you can, but I found how to do that later….
When run in an shell or Xwindow, the script above appears to hang because it reads the password from /dev/tty. The /dev/tty that the script is looking for belongs to the shell or Xwindow that started the process, so the script waits for keyboard input.
First Working Script
#!/bin/sh
HOST='remote.host'
USER='theirname'
PASSWD='theirpw'
FILE='file.txt'
ftp -n $HOST < <END_SCRIPT
quote USER $USER
quote PASS $PASSWD
put $FILE
quit
END_SCRIPT
exit 0
Getting the password to the ftp server without having the ftp client program read the password from /dev/tty requires two tricks:
- Using the -n option on the ftp client program to prevent the ftp client from trying to log in immediately. That way, the ftp client does not ask for a user ID and password. No use of /dev/tty.
- Use the ftp client program command quote to pass the user ID and password to the ftp server.
Further Refinements:
Even if everything works perfectly, the user running the above script will see lots of text scrolling by. One refinement is to redirect output to different places:
ftp -n $HOST > /tmp/ftp.worked 2> /tmp/ftp.failed < <END_SCRIPT
One could further refine error handling by acting on the ftp client program’s exit status:
ftp -n $HOST > /tmp/ftp.worked 2> /tmp/ftp.failed < <END_SCRIPT
blah blah
END_SCRIPT
EXITSTATUS=$?
if [ $EXITSTATUS != "0" ]
then
# handle the error...
fi
The problem is that most FTP clients exit with a status of 0; any error is usually an error relating to an individual command (such as cd to a nonexistent directory, and is reported as such). This leads to ugly situations: the file transfer fails, but the script doesn’t detect the problem.
One way to verify that a file transfer took place is to transfer it back.
#!/bin/sh
ftp -n < <END_SCRIPT
open $1
user $2 $3
put $4
get $4 retrieval.$$
bye
END_SCRIPT
if [ -f retrieval.$$ ]
then
echo "FTP of $4 to $1 worked"
rm -f retrieval.$$
else
echo "FTP of $4 did not work"
fi
echo "ftp.scr qaserver user passwd /db2/PRD/log_dir/S1234567.LOG" | at now
two issues here:
- FTPs there and back of large files can consume a lot of time.
- If the original FTP only partially transfers the file (due to, say, a space problem on the target), the subsequent FTP may still run, overwriting the original file.
Control of ftp by a shell script
One obvious improvement would have the ftp client program controlled by the shell script. I don’t think that would comprise an impossible task, but I also don’t think that it would have much value.
Alternative #1
Linux, Unix and BSD users have the alternative of using a .netrc file. The ftp man page documents the format of .netrc. The example below allows FTP to two different servers.
machine qaserver
login qauser
password qapasswd
machine devserver
login devuser
password devpasswd
For security, ftp requires that the .netrc file not have group or world read or write permissions:
$ ls -l .netrc
-rw------- 1 menglis3 users 51 Oct 16 13:30 .netrc
Using a .netrc file has a few problems that may or may not prevent you from using it.
- A shell script that does FTP using .netrc is no longer self-contained. You have to keep track of two files, which means that bugs can be less than obvious.
- ftp reads the .netrc of the user it is running under. If you develop your script under a given user ID, then put it in production under a second user ID, you have to coordinate .netrc file contents between those two user IDs.
Topics: Code, Technology, Work | 1 Comment »
October 17th, 2006 at 11 36 pm
ITs ok
but in case of giving input parameters
FTP method : put or get
in such cases
error handling is getting only for get file
what abt put file
if any answer plz mail to me
i will be thankful to you