↓Skip to main content

Linux Filetransfer and Backups: SCP, SFTP, Rsync, Rsnapshot, dd, Borg Backup & Netcat

1802 words·
Linux Filetransfer Backup Commands
Table of Contents

Prerequisites
#

Make sure to establish an SSH connection to the remote server from where the data is backed up, before starting a backup.

# Create SSH Key pair
ssh-keygen -t rsa -b 4096

# Add SSH Key to remote server
ssh-copy-id -i ~/.ssh/id_rsa.pub user@IP

# Establish Connection / confirm the hostkey & create a trusted relationship
ssh user@IP

SCP
# 

# Copy file to remote location
scp /path/file user@IP:/path/destination

# Copy folder to remote location
scp -r /path/folder user@IP:/path/destination

# Copy folder to remote location: Preserve permissions, not ownership
scp -rp /path/folder user@IP:/path/destination

# Copy folder to remote location: Define custom port
scp -P 2222 /path/folder user@IP:/path/destination


# Copy file from remote location
scp user@IP:/path/file /path/destination

SFTP
#

This tutorial enables SFTP without shell access for the SFTP users. SSH File Transfer Protocol is a secure way to transfer files using an SSH connection.

Setup
# 

# Install dependencies
sudo apt install openssh-server

# Create user for sftp access
adduser user1

# Create directory for user
mkdir -p /sftp/files/user1

# Set owner to root
chown root:root /sftp/files

# Read & execute permission for u & o
chmod 755 /sftp/files

# User must own his folder
chown user1:user1 /sftp/files/user1

Note: The partent directories of the user folder must be owned by root and not writable by other users.

Add the following configuration to vi /etc/ssh/sshd_config, it only applies to the specified user:

# SFTP for user1
Match User user1
ForceCommand internal-sftp
PasswordAuthentication yes
ChrootDirectory /sftp/files
PermitTunnel no
AllowAgentForwarding no
AllowTcpForwarding no
X11Forwarding no

Parameters

  • ForceCommand internal-sftp Run SFTP, disallow shell access

  • ChrootDirectory /sftp/files Restricts access to defined directory

# Restart SSH daemon
sudo systemctl restart sshd

SFTP Commands
#

  • Connect / disconnect
# Connect to SFTP server
sftp user1@IP

# Connect to SFTP server: Custom port
sftp -oPort=2222 user1@IP

# Exit the connection
exit

# List SFTP commands
help
  • File download
# Download file
get filename

# Download file to specific dir
get filename /local/dir/
  • File upload
# Upload file (From current local dir)
put filename

# Upload file: Upload file to specific dir
put filename remote/dir/
  • Local directory
# List local dir
lls /local/dir/

# Change local dir
lcd /local/dir

# List current local dir
lpwd

# Create local directory
lmkdir dirname

Rsync
#

Syntax
#

  • Test run
# Copy folder to remote location: Test run
rsync -azP --dry-run -e ssh /path/source user@IP:/path/destination
  • Production
# Copy folder to remote location
rsync -azP -e ssh /path/source user@IP:/path/destination

# Copy only files that start with "name" from folder to remote location
rsync -avP --include 'name*' --exclude '*' /path/source/ user@IP:/path/destination


# Copy folder from remote location
rsync -azP -e ssh user@IP:/path/source /path/destination

# Copy folder from remote location (custom SSH port)
rsync -azP -e "ssh -p 2222" user@IP:/path/source /path/destination

# Preserve numerical user and group IDs
rsync -avzh --numeric-ids -i -e ssh root@10.8.150.37:/source/path /target/path

# Preserve numerical user and group IDs, delete files from target
rsync -avzh --del --numeric-ids -i -e ssh root@10.8.150.37:/source/path /target/path

Options
#

  • -a Preserves the permissions, ownership, timestamps and other attributes

  • -z Use compression during the transfer

  • -v Verbose: list each file that is transferred

  • -P P-partial and –progress

  • --partial Rsync will not delete the partially transferred file and will continue the transfer from where it left off on the next run

  • --progress Display the progress of the transfer, including the amount of data transferred, the transfer speed, and the estimated time remaining

  • -e ssh Specifies the remote shell to use: SSH

  • --numeric-ids Preserves the numerical user and group IDs rather than mapping them by name

  • -i Outputs a change-summary for all updates

  • --del Delete files in target directory

  • --dry-run Used to for testing, lists the files that would be synced

Rsync Troubleshooting
#

Unix limitation of command argument memory maximum size, Error Code:

bash: /usr/bin/rsync: Argument list too long 
rsync: connection unexpectedly closed (0 bytes received so far) [Receiver]
rsync error: remote command could not be run (code 126) at io.c(235) [Receiver=3.1.2]

Solution:
Don’t copy the files from /source/directory/*, copy the whole directory instead /source/directory


Rsnapshot
#

Installation
# 

# Update Package Manager
sudo apt update

# Install Rsnapshot
sudo apt install rsnapshot -y

Configuration
#

Open the Rsnapshot Configuration file:

sudo vi /etc/rsnapshot.conf
Note: Be sure to always use “tabs” and not “spaces” between conig parameters!

Define the directory for the Rsnapshot backups:

snapshot_root   /backups/

Enable SSH and define SSH Key:

# Enable remote ssh backups
cmd_ssh        /usr/bin/ssh

# Define SSH Key
ssh_args        -i /home/ubuntu/.ssh/id_rsa

Define Backup Level / Interval

# This creates a 3 day backup rotation
retain  daily   3

Define Lockfile Path: The user that runs Rsnapshot must have the necessary permissions to write to the lockfile. The lockfile is used to ensure that only one instance of Rsnapshot is running at a time.

lockfile        /home/ubuntu/rsnapshot.pid

Define Backup
#

Define Remote Backup

# Syntax
backup  user@IP:/data/path     destination

# Example
backup  ubuntu@192.168.30.155:/home/ubuntu     backup1

Test Configuration
#

Make sure to test the Rsnapshot Configuration!

# Test Rsnapshot Configuration
sudo rsnapshot configtest

# Shell Output:
Syntax OK

Trigger Backup
# 

# Rsnapshot is usually triggered with a crontab
0 2 * * * /usr/bin/rsnapshot daily

# Manually trigger Backup: 
rsnapshot daily # or
/usr/bin/rsnapshot daily

Backup Location
#

Following my example, the backups are located in the following directory:

/backups/daily.0/backup1
/backups/daily.1/backup1
/backups/daily.2/backup1

DD Dublicate Data
#

The dd command is a usful tool for creating backups of disk partitions. In addition to backing up individual partitions, it can be used to create a backup of the entire Linux hard drive. This allows to capture the complete state of the disk, including all partitions, boot records, and file systems.

Common Options

  • if Input file / source

  • of Output file / destination

  • bs Define the blocksize (512 bytes default to a maximum of 64M)

  • status Shows the progress of the dd command

  • conv=notrunc dd overwrites an existing destination file per default. This can be stopped with this option.

  • conv=append Append data to an existing destination file

Disk Partition Backup
#

Note: If it’s necessary to create a hot backup of a partition, create a LVM snapshot (if possible) before running the dd command.

Local Backup:

# Create backup
sudo dd if=/dev/sda1 of=/path/to/backup.img status=progress

# Restore backup
sudo dd if=/path/to/backup.img of=/dev/sda1 status=progress

Remote Backup:

# Create local backup of remote server partition
ssh user@IP "sudo dd if=/dev/sda1" | sudo dd of=/path/to/local-backup.img

# Create local backup of remote server partition: Show progress
ssh user@IP "sudo dd if=/dev/sda1" | sudo dd of=/path/to/local-backup.img status=progress


# Create backup of local partition and save it to a remote server
sudo dd if=/dev/sda1 | ssh user@IP "sudo dd of=/path/to/remote-backup.img"

# Create backup of local partition and save it to a remote server: Show progress
sudo dd if=/dev/sda1 | ssh user@IP "sudo dd of=/path/to/remote-backup.img" status=progress

Borg Backup
#

Installation
# 

# Update Package Manager
sudo apt update

# Install Borg Backup
sudo apt install borgbackup -y

Initialize Repository
#

Initialize a directory where the the Backup Archives are stored as Borg Repository.

# Create directory for backup repository
mkdir borg-repository


# Initialize backup repository: Without encryption
borg init -e none ~/borg-repository

# Initialize backup repository: Repokey encryption key
borg init --encryption=repokey ~/borg-repository

# Initialize backup repository: Keyfile encryption key
borg init --encryption=keyfile ~/borg-repository

When an encryption key is used a passphrase for the keyfile must be defined, Shell Output:

Enter new passphrase: 
Enter same passphrase again:
Do you want your passphrase to be displayed for verification? [yN]:

IMPORTANT: you will need both KEY AND PASSPHRASE to access this repo!
If you used a repokey mode, the key is stored in the repo, but you should back it up separately.
Use "borg key export" to export the key, optionally in printable format.
Write down the passphrase. Store both at safe place(s).

Encryption Keys:

repokey The encryption key will be stored in the repository configuration file
keyfile The encryption key will be stored in ~/.config/borg/keys

Note: The encryption mode can’t be changed later. It is necessary to provide the repository encryption passphrase every time a backup archive is created or listed.

# If necessary the passphrase can be changed as follows
borg key change-passphrase ~/borg-repository

Create Backup
# 

# Create Backup Archive: Syntax
borg create ~/borg-repository::backupname-day1 /source/path
borg create ~/borg-repository::backupname-day2 /source/path
borg create ~/borg-repository::backupname-day3 /source/path
...

# Verbose Output: List files
borg create --list ~/borg-repository::backupname /source/path


# Exclude path from source
borg create ~/borg-repository::backupname /source/path --exclude /dir1 /dir2

Remote Backup

Install and run Borg Backup on the host with the data that should be backed up and copy it via SSH the Borg Backup Server / the Borg Repository:

borg create ssh://user@backup_server/~/borg-repository::backupname /path/to/data

List Backup Archives & Data
# 

# List all Backup Archives in the repository
borg list ~/borg-repository

# List Conect from Backup Archive
borg list ~/borg-repository::backupname

Restore Backup
# 

# Extract Backup Archive in current working directory
borg extract ~/borg-repository::backupname

# Verbose Output
borg extract --list ~/borg-repository::backupname

Strip Path

By default Borg Backup extracts the backup with the original full path from the backed up data. To remove elements from the path use --strip-components and define the number of path elements to strip. For example --strip-components 3 would strip the first 3 path components.

# Strip Components
borg extract --strip-components 3 --list ~/borg-repository::backupname

Mount Backup
# 

# Create directory for Backup Archive mount
mkdir ~/borg-mount

# Mount Backup Archive
borg mount ~/borg-repository::backupname ~/borg-mount

# Unmount Backup Archive
borg umount ~/borg-mount

Delete Backup Archive
# 

# Delete Backup
borg delete ~/borg-repository::backupname-day

Netcat
#

Netcat is a versatile utility that reads and writes data across network connections using the TCP/IP protocol.

# Netcat commands
nc -h

Filetransfer
#

Receiving server
# 

# Receiving server: Define a listening port and write the incoming data to a file
nc -l -p 12345 > received_file

Sending server
# 

# Sending server: Leave connection open after the file was sent
nc receiver-IP 12345 < file_to_send

# Sending server: Close nc after fransfer has finished
nc -q 0 receiver-IP 12345 < file_to_send

Show transfer progress

# Install PipeViewer
sudo apt install pv

# Sending server: Use PipeViewer to show the transfer progress
pv file_to_send | nc receiver-IP 12345

# Sending server: Close nc after fransfer has finished
pv file_to_send | nc -q 0 receiver-IP 12345

  • -l Listen for an incoming connection

  • -p Specifies the port on which to listen (choose any unused port)

  • > Redirects the data received by netcat to a file

Port Scanning
# 

# Port scanning: Syntax
nc -zv target-IP port_range

# # Port scanning: Ignore DNS lookup on the IP
nc -zvn target-IP port_range

# Port scanning: Example
nc -zv 192.168.30.60 80 443

Chat Server
# 

# Server 1
nc -l -p 12345

# Server 2
nc server-IP 12345

Once the connection is established, text typed into the terminal on either server will appear on the other server. This way, two users can chat with each other using simple text messages.