networking - How to Write a Shell Script to Move Files from FTP Server to Network Share Drive

19
2014-04
  • nicorellius

    I have customers who upload large amounts of data to my FTP server (an Ubuntu 9.10 machine). Once the files are uploaded, I am faced with transferring them to a local Droboshare network drive so our technical support staff can retrieve them and analyze the issue.

    After I mount the network share at /media/<sharename> I run this command manually to get these files:

    sudo mv /home/ftp/<data_dir>/<file_name> /media/<network_share>/<ftp_user_data>

    but this takes forever and if there are tens or hundreds of files, I can't realistically do this over and over again. I thought it would be nice to have a script that I could run periodaclly to transfer a group of these files to the network share. I can sort them as the technical support staff uses them.

    I am still new to Linux and writing shell scripts. Anyway to do this easily? Just to add to the description here, the files could be ZIP files, FASTA files, TAR.GZ files, and/or TXT files. Also, if the ZIP files are large, certain zipping programs convert these to ZIP.001, ZIP.002, ZIP.003, etc... So the file type in this FTP directory could be quite varied.

    I was thinking the way I can identify these files is by simply transferring ALL files in the directory (although this may take a while) or to somewhat designate a subset of these, according to time completed or something like this.

    I'm open to ideas. Thanks in advance.

  • Answers
  • Marco Ceppi

    Why not just make the FTP folder the actual mount point for the network drive? I run into a similar issue where I'm constantly backing up VirtualMachines on a Linux machine to a Windows network share (since the majority of the company infrastructure is Windows). This is my structure:

    /media/windows-share is my mount point

    For continuity I've created a symlink in my backup application:

    /opt/backup/mnt so if the mount point changes in the future my program doesn't need an update.

    Lastly I employ a cool little tool called autofs (sudo apt-get install autofs) That guide is decent (and up to date) - though I've employed this with a slightly different approach on my blog.

    In your case I would create a symlink where my FTP dropbox is (/home/ftp/<data_dir>/) to the appropriate network folder in /media/<network_share>/<ftp_user_data> that way there's no wasted time in transferring across disks and the files are available immediately.

  • msw

    For duplicating files from here to there (or there to here), rsync is a swiss army chainsaw which will likely do what you want.

  • ℝaphink

    You could try to process files as soon as they're uploaded to the FTP, using a technology sucha s inotify. inoticoming is a tool you can use for that, see http://manpages.ubuntu.com/manpages/lucid/man1/inoticoming.1.html.


  • Related Question

    command line - How do I write an application install shell script?
  • jumpnett

    Does anyone know how to write a shell script to install a list of applications? It's a pain to have to install each application by hand every time I set up a new system.

    Edit: It still asks me Do you want to continue [Y/n]?. Is there a way to have the script input y or for it not to prompt for input?


  • Related Answers
  • qbi

    I would assume the script would look something like this:

    #!/bin/sh
    apt-get update  # To get the latest package lists
    apt-get install <package name> -y
    #etc.
    

    Just save that as something like install_my_apps.sh, change the file's properties to make it executable, and run it from the command line as root.

    (Edit: The -y tells apt-get not to prompt you and just get on with installing)

  • Xiao-Long Chen

    Just create a list of apps in a file, example.list, and run

    cat example.list | xargs sudo apt-get -y install
    
  • NlightNFotis

    I would opt for the following script: vim install

    #!/bin/bash
    apt-get update  # To get the latest package lists
    apt-get install $1 -y
    

    Then I should make the above script executable chmod +x install. Then to use it, I could type: ./install <package_name>. Example: ./install clang