Files can be easily copied from one location to another location in a machine by using “cp” command. However things gets complex when you need to copy files from one machine to another machine. Obviously, your favorite command “cp” is not going to work!

The best solution would to use “rsync” which would transfer the file efficiently. Again, this will not work if the rsync daemon is not configured properly or you do not have ssh access.

The other solution which I used is “wget“. Since wget can download files using HTTP/HTTP/FTP, it seems to be an good solution for me. And Yes, It was!

A simple example of copying directory from remote machine would be

wget -r ftp://<USERID>:<PASSWORD>@<MACHINE-NAME>//home/test

The above command would copy the content (and obviously -r denotes recursively) of /home/test to the local directory ./<MACHINE-NAME>/home/test

Now lets say, you would like to copy the content of /home/test to some other location (for example, say /test/content/data . Following commands will help in doing so.

REMOTE_LOCATION=/home/test
LOCAL_LOCATION=/test/content/data

wget -r --no-host-directories --force-directories \
--cut-dirs=`echo $REMOTE_LOCATION | awk -F"/" '{ print NF }'`\
 --directory-prefix=$LOCAL_LOCATION \
ftp://<USERID>:<PASSWORD>@<MACHINE-NAME>/$REMOTE_LOCATION

Another thing to mention here is, by default, wget would traverse only five level down the directory hirerachy and hence can copy them accordingly. So if you have files in directory structure as /home/test/data/xyz/content/resources/site1/css, contents of that would never be copied by wget. If you need to do so, you should specify depth level also in your command as

REMOTE_LOCATION=/home/test
LOCAL_LOCATION=/test/content/data

wget -r --level=9 --no-host-directories --force-directories \
--cut-dirs=`echo $REMOTE_LOCATION | awk -F"/" '{ print NF }'` \
--directory-prefix=$LOCAL_LOCATION \
ftp://<USERID>:<PASSWORD>@<MACHINE-NAME>/$REMOTE_LOCATION

wget is nice (specially when you do not have the options to use rsync or rcp/scp). Try it!

Complete options for wget can be browsed at http://www.gnu.org/software/wget/manual/wget.html

Tags: copy, ftp, rsync, script, shell, Unix, wget