Data transfer: linux
Note: When the amount of data to transfer exceeds around 100 GB, methods like scp, sftp, rsync may be too slow, and Globus will be faster for transferring collections of files due to doing so in parallel.
- Secure copy or SCP is a means of securely transferring computer files between a local host and a remote host or between two remote hosts.
- It is based on the Secure Shell (SSH) protocol
- Normally, a client initiates an SSH connection to the remote host, and requests an SCP process to be started on the remote server. The remote SCP process can operate in one of two modes:
- source mode, which reads files (usually from disk) and sends them back to the client
- sink mode, which accepts the files sent by the client and writes them (usually to disk) on the remote host
scp source_directory/hello_world.txt destination_directory
scp source_directory/hello_world.txt username@remote_host:destination_directory
- sftp is a command-line interface client program to transfer files using the SSH File Transfer Protocol (SFTP) as implemented by the sftp-server command by the OpenSSH project, which runs inside the encrypted Secure Shell connection.
- It provides an interactive interface similar to that of traditional FTP clients
- should not be confused with running an FTP client over an SSH connection.
- Gets txt file from Big Horn to Local System:
get hello_world.txt destination_directory
- Puts txt file from Local System on to Big Horn:
put source_directory/hello_world.txt destination_directory
- rsync is a utility to keep copies of a file on two computer systems
- functions as both a file synchronization and file transfer program
- The rsync algorithm is a type of delta encoding, and is used to minimize network usage. Zlib may be used for additional compression, and SSH or stunnel can be used for data security
- Rsync is typically used to synchronize files and directories between two different systems.
- For example, if the command rsync local-file user@remote-host:remote-file is run, rsync will use SSH to connect as user to remote-host
- To sync the contents of dir1 to dir2 on the same system:
rsync -r dir1/ dir2
- To sync with a remote system:
rsync -a ~/dir1 username@remote_host:destination_directory