Home-made PC Router and NAS Server [Page 4] - NAS

NAS build

The second objective of the build is a host for my files. As mentioned, this is not an enterprise grade NAS, but it has some useful features and is great for the home user.

My setup uses two hard drives, now upgraded to Seagate 2TB disks, in 2.5 inch size.

My intention is to ensure that the storage is resilient and recoverable in two potential events:

  1. If one of these hard drives fails, I can get everything from the other one
  2. If I do something silly like overwrite or delete files, to be able to restore them from the previous day or further back

When I initially did this setup, I did just create one XFS partition on each drive and used regular RSYNC commands to copy the changes over to the backup drive. This is ideal and simple.

But nowadays BTRFS and ZFS file systems work well and stable on Linux. Both of these are advanced and support snapshots and RAID. ZFS is more enterprise grade (having come from Sun, now Oracle), but BTRFS seems to have more community information and having never tried it, I decided to have a go on my previous 16.04 build. Since it worked well, in this build I will continue to use it. Here is my guide, with some refinements since:

Assuming you have two new disks for the NAS, you will be either:

  1. Creating a fresh file system on two new disks, and copying any file onto it via the network, after creating the file systems
  2. Re-using the disks but changing from one file system to BTRFS and retaining the data
  3. Upgrading the disks but because you have only one spare SATA port, adding the second disk later
  4. Upgrading the disks and you have enough SATA ports to add both disks in

Option 1 is easiest, but the copy will be time consuming and could be unreliable with huge volumes.

Option 2 was my first scenario when I upgraded from XFS to BTRFS on my 1TB disks. Option 3 is my latest upgrade path since I'm moving to 2TB disks.

Creating the partition (fdisk)

In all cases, we will need to create a partition on at least one disk first.

The utility fdisk will list your drives:

sudo fdisk -l

You may see some /dev/loop disks - these can be ignored. Your hard drives and SSDs will be named /dev/sd_

Partitions will appear as numbers after /dev/sd_. E.g. my first disk (120G SSD) is connected to port 0 and is named /dev/sda. It has 5 partitions on it e.g. /dev/sda1 for the first partition.

Disk /dev/sda: 111.8 GiB, 120034123776 bytes, 234441648 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: A9F1CA66-D5A4-4E62-95B8-284D623B1C6D

Device        Start       End   Sectors Size Type
/dev/sda1      2048      4095      2048   1M BIOS boot
/dev/sda2      4096   4198399   4194304   2G Linux filesystem
/dev/sda3   4198400  20975615  16777216   8G Linux swap
/dev/sda4  20975616  41947135  20971520  10G Linux filesystem
/dev/sda5  41947136 209719295 167772160  80G Linux filesystem


Disk /dev/sdb: 1.8 TiB, 2000398934016 bytes, 3907029168 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes


Disk /dev/sdc: 931.5 GiB, 1000204886016 bytes, 1953525168 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disklabel type: dos
Disk identifier: 0x000a16eb

Device     Boot Start        End    Sectors   Size Id Type
/dev/sdc1        2048 1953523711 1953521664 931.5G 83 Linux


Disk /dev/sdd: 931.5 GiB, 1000204886016 bytes, 1953525168 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disklabel type: dos
Disk identifier: 0x0005eb02

Device     Boot Start        End    Sectors   Size Id Type
/dev/sdd1        2048 1953523711 1953521664 931.5G 83 Linux

My new 2TB drive is named /dev/sdb since it is on the second SATA port on the motherboard. My old 1TB drives are /dev/sdc and /dev/sdd.

Before we can create a BTRFS file system on the disk, we should partition it first. To do this, run:

sudo fdisk -c /dev/sdb

If you are in scenario 2 (reusing two existing disks), you will want to delete one disk and re-partition it for use with BTRFS. Press 'p' to print the current partitions, and if any are listed, press 'd' to delete them. When complete, press 'w' to write the changes.

It's more modern to use the GPT partition scheme now for disks, rather than the legacy MSDOS scheme. To do this, press 'g'.

To create a new partition, press 'n'. Accept the following three defaults for partition number (1), first sector (2048), last sector (whatever the last sector on your disk is).

Press 'p' to verify the partition information is correct, then 'w' to write changes.

Now refresh the system partition information, this command does this well:

sudo partprobe -s

Creating the file system

We can now move on to creating our BTRFS file system.

We will start by creating the partition on only one disk, as my scenarios 2 and 3 above. For scenario 4, I also recommend this method of bulk copying data to one disk first, and then balancing to the second disk later.

My first BTRFS storage will be created on disk /dev/sdb and I'll copy files from the old disk on /dev/sdc to this (scenario 3).

The partition I am creating the new BTRFS on is sdb1:

mkfs.btrfs -L MASTER /dev/sdb1

And it's done. Let's mount it. I'll mount it in new folder /mnt/master but you can adjust as you need:

The mount option below has specified ZSTD for compression. This is new and performs better than LZO or ZLIB. It's not too CPU intensive, so should give me some performance boost, however during the copy on my Quad Core i5 3475S 2.9GHz CPU, I was seeing about 45% usage across all cores (of 400%, on one core about 22% usage) so if you are running a low performance CPU such as Intel Atom, older Celerons or older AMDs, you may want to avoid compression.

Over my data, the compressed result was 824GB out of 900GB of data, so some saving. You're results may be different. I have lot of files that are pictures and videos, so they are already compressed (e.g. JPEGs) and they'll be no space gain on these.

I've also specified noatime as an option - this specifies not to write access times for files and directories (created/modified dates are still written).

sudo mkdir /mnt/master
sudo mount -o compress=zstd,noatime /dev/sdb1 /mnt/master

A quick command 'df -h' and we can see /dev/sdb1 is mounted in /mnt/master.

I then create a sub volume for HOME. This is where I will store all my files:

sudo btrfs subvolume create /mnt/master/home
sudo btrfs subvolume list -p /mnt/master

And it's ready. Now to copy all my files.

In my earlier scenario 2, I mounted the remaining XFS partition so I could copy from it:

sudo mkdir /mnt/prior
sudo mount /dev/sdc1 /mnt/prior

In scenario 3 that I'm in now, my old 1TB BTRFS disks were already mounted in /mnt/main

Before copying, I want my own user to own the directory (not root) so adjusted the ownership permission:

sudo chown -R dan /mnt/master

Then this is the copy command:

nohup cp -ax /mnt/main/home/* /mnt/master/home &

Run that and it will take a while to copy almost a terabyte. I've prefixed the cp command with 'nohup ' and added ampersand '&' at the end to ensure the copy runs in the background, not via SSH, in case it times out. Then, let it run for hours, and continue tomorrow! Check nohup.out for any errors or messages (they'll be no output at all if successful).

To check it is working, use tail nohup.out. This should show no errors if it is working.

You can also install iotop to check disk usage. It's not installed by default, so run 'sudo apt install iotop' and once installed, use 'sudo iotop' to see the usage (needs to run as root).

Once complete, check space usage old vs new, there may be some change expected due to the different file systems, or compression being taking effect.

Then do some spot checks just to make sure.

df -h
du -sh /mnt/master/home

The df command will show the total usage of the disk (so the compressed size). The du command will sum all the files and give the uncompressed size (this will take a while).

Creating the RAID volume

Now that all files have copied over (do a check first!), we can use the second disk to create a software RAID1 setup.

This will replicate the first disk on to the second.

Before starting, un-mount the existing mount that you copied from earlier. Remove any disks you no longer want and add the second new disk.

If re-using an existing disk, delete existing partitions and recreate the single partition again using the exact same options as before.

The 'btrfs device add' command will create the new file system if you have already partitioned. If the disk is new, create a partition like before detailed above. Here I'm creating on the /dev/sdc1 partition.

sudo btrfs device add /dev/sdc1 /mnt/master

The balance command will then spread data and metadata to the second disk.

The options dconvert and mconvert state that we are going to use both data and metadata in RAID1.

nohup sudo -b btrfs balance start -dconvert=raid1 -mconvert=raid1 /mnt/master

Again I'm using nohup to log output and let the command run in the background (with the sudo -b option). Before running, if you have a previous nohup.out file, delete it first so you're not confused by old output (otherwise the output of this command will just append to the existing file)

After a long while (due to the amount of data like before), this will replicate all the data onto the second disk. Come back hours later or another day!

Once complete, we now have a working file system, with all files copied over and it is resilient to a single disk failure.

The last thing we need to do is make the mount permanent so that the BTRFS raid setup is available in /mnt/master/home

Use these commands to check the UUID of the disks (changing sdb1/sdc1 as appropriate to your setup):

sudo btrfs filesystem show /mnt/master/
blkid /dev/sdb1
blkid /dev/sdc1

This should list volumes with the same LABEL, UUID but different UUID_SUB. It is UUID that we actually want to use.

Now edit /etc/fstab

sudo nano /etc/fstab

Here is my example, yours will vary. The line at the bottom is the added mount. Note that I am mounting the whole volume rather than any subvolumes (unlike before). This is because the 'compress' mount option does not work for subvolumes.

# /etc/fstab: static file system information.
#
# Use 'blkid' to print the universally unique identifier for a
# device; this may be used with UUID= as a more robust way to name devices
# that works even if disks are added and removed. See fstab(5).
#
# <file system>                           <mount point>   <type>  <options>                      <dump> <pass>
UUID=7e586b39-aa00-11e8-94e6-e46f13a8c8a0 none            swap    sw                             0      0
UUID=7fb20af2-aa00-11e8-94e6-e46f13a8c8a0 /               ext4    defaults,noatime               0      0
UUID=7e586b38-aa00-11e8-94e6-e46f13a8c8a0 /boot           ext4    defaults,noatime               0      0
UUID=828c03b8-aa00-11e8-94e6-e46f13a8c8a0 /mnt/ssd        ext4    defaults,noatime               0      0

# Main hard disks:
UUID=70a651ab-4837-4891-9099-a6c8a52aa40f /mnt/master     btrfs   defaults,noatime,compress=zstd 0      0

Reboot the system and see if it works:

df -h
Filesystem      Size  Used Avail Use% Mounted on
udev            3.9G     0  3.9G   0% /dev
tmpfs           791M   30M  762M   4% /run
/dev/sda3        20G  5.6G   14G  30% /
tmpfs           3.9G  232K  3.9G   1% /dev/shm
tmpfs           5.0M     0  5.0M   0% /run/lock
tmpfs           3.9G     0  3.9G   0% /sys/fs/cgroup
/dev/sda3        20G  5.6G   14G  30% /home
/dev/sda2       277M  154M  110M  59% /boot
/dev/sdb1       299G  116G  181G  40% /mnt/software
/dev/sda4        67G   14G   54G  21% /mnt/ssd
/dev/sdd1       932G  802G  130G  87% /mnt/master/home
tmpfs           791M   80K  791M   1% /run/user/1000

Network Disk Share

Samba is what is used to make the hard drive appear on the network. It's basically compatible with Windows file shares without requiring extra software.

To install it:

sudo apt install samba nfs-kernel-server
sudo nano /etc/samba/smb.conf

Change some things from the default. Firstly, add some settings below [global]

security = user
name resolve order = hosts wins bcast
wins support = yes
force user = dan
server min protocol = SMB3
client min protocol = SMB3
client max protocol = SMB3

These settings setup user security, make Samba a WINS (Windows Internet Naming Service) server too, and prioritises WINS over bcast for name resolution (this improves the speed of the share). As SMB1 and SMB2 are old (SMB1 now removed from Windows 10), Samba has also be configured to use SMB3 as the minimum protocol.

One thing we'll also want to do is make sure samba only listens on the LAN network, and only allows connections from localhost (127.0.0.1) and LAN IP ranges (10.0.0.0/8):

hosts allow = 127.0.0.1 10.0.0.0/8

interfaces = enp2s0

bind interfaces only = yes

The IP 10.0.0.0/8 is specified instead of 10.0.1.0 because our VPN network (created later on 10.8.0.0) will also be allowed to connect.

Add the shares that you need:

[share]
comment = This share requires a password
path = /mnt/master/home
browsable = yes
guest ok = no
writable = yes
read only = no
create mask = 0755
user = dan

[ssd]
comment = This share requires a password
path = /mnt/ssd
browsable = yes
guest ok = no
writable = yes
read only = no
create mask = 0755
user = dan

Finally, run:

sudo smbpasswd -a dan

Enter the password the same as the Linux O/S one. Without doing this, I got access denied errors when trying to connect.

I've given two examples - one share is for my NAS drive subvolume (mounted in /mnt/master/home), the other is for a portion of my SSD. Both shares can only be accessed by the 'dan' user.

To add another user, do:

sudo useradd user2

sudo passwd user2

sudo smbpasswd -a user2

Once done, run:

sudo service smbd restart

Now you should be able to go to a Windows PC and browse to the IP address of your server we configured earlier, e.g. \\10.0.1.2

.

Once you go into a share, you should be prompted for a username and password. You could, if you wish, create a Linux user with the same login name and password as your Windows one and it should automatically pass through.

Performance:

Samba performance is not that great by default. NFS may be a better way to go, but my experiments proved less quick and it was more complex to setup (and can only be accessed by 'Professional' editions of Windows). Adding the following to /etc/samba/smb.conf helped performance quite a lot:

 

# Performance options:
server signing = disabled
read size = 65536
read prediction = true
write raw = yes
read raw = no

After applying, restart using:

sudo service smbd restart

Backups / Snapshots

RAID 1 protects us from disk failure, but it does not protect us from mistakes!

If I do something like delete or overwrite files (probably by accident), RAID 1 automatically replicates that mistake to the second disk, and the original file is gone forever.

BTRFS support snapshots however. It uses a technology called Copy-On-Write.

This means that Snapshots are not the size of the entire disk each time. The first snapshot will actually be nothing in size, but as I delete and overwrite files, these old ones will be linked to the snapshot.

I can then use that snapshot to recover the files.

Over time, files that you deleted or overwrote will consume space if linked to old snapshots, so manage the amount of snapshots carefully. Snapshot changes will be by block size instead of file size.

Snapshots need to be of a subvolume and saved to a subvolume, so we need to create a subvolume for snapshots first.

sudo btrfs subvolume create /mnt/master/snapshots
sudo btrfs subvolume list -p /mnt/master

To take a snapshot, use this command:

sudo btrfs subvolume snapshot -r /mnt/master/home/ /mnt/master/snapshots/snap20170305

Once the snapshot has been taken, You can have a look at your snapshot to see the files.

ls /mnt/master/snapshots/snap20170305

Let's make a change:

rm /mnt/master/home/snap20170305/dan/Downloads/install.exe

The file is gone. To restore a previous version or deleted file, you could just copy it from your snapshot (or mv to remove it from the snapshot to the main volume):

cp -a /mnt/master/home/snap20170305/dan/Downloads/install.exe /mnt/master/home/dan/Downloads

The file is back!

If you delete many files, and just want to restore the entire snapshot, you can run these commands instead:

rsync -avrW /mnt/master/home/snap20170305/* /mnt/master/home

You may want to then delete snapshots to save space

sudo btrfs subvolume delete /mnt/master/home/snap20170305

This is all good, but snapshots are not automatically taken - you need to take them first.

A crontab script can help with this.

Crontab is frequently used for scheduling tasks in Linux, so we can use it to execute a script that will take snapshots (and clean up old ones) at 2AM daily.

First, the custom script:

nano ~/snapshots.sh
#!/bin/bash

TIME=`date +%Y%m%d`
OLDTIME=`date +%Y%m%d  -d "7 days ago"`
HOMENAME=/mnt/master/home
SNAPSHOTNAME=/mnt/master/snapshots/home$TIME

# Take a new snapshot today
btrfs subvolume snapshot -r $HOMENAME $SNAPSHOTNAME

# Delete snapshots older than 7 days ago, matching the name of these snapshots
find /mnt/master/snapshots/* -maxdepth 0 -print | awk -v b=$OLDTIME '{a=substr($0,length($0)-7,8); if (a < b) print "/mnt/master/snapshots/home"a; }' | xargs btrfs subvolume delete -c

Make it executable.

sudo chmod +x ~/snapshots.sh

Now add it to crontab.

sudo crontab -e

Add the following line as an example to call the script daily at 2AM:

0 2 * * * /home/dan/snapshots.sh

To explain the script above - we are assigning the date to a variable, and a date from 7 days ago to another variable. A snapshot is created for today's date - easy.

Deleting old snapshots (since you may not want them around forever - disk space will get lost) was trickier. You cannot reliably use the directory time stamps of the snapshots (they match the main filesystem), so we match the filename pattern instead.

The find command is used to match directories in /mnt/master/home starting with home. Maxdepth prevents going into sub directories. Awk then extracts the last 12 characters of the file name and tests if it is less than or equal to the file name 7 days ago (prefixed with home). If true, the full path is printed and put to btrfs subvolume delete with xargs.

Using this method, I've got backups for the last 7 days, but I can also take manual snapshots in the same directory still and they will not be deleted, as long as I don't suffix with a date format in the last 12 characters.

References and more reading:
BTRFS Wiki
HowToForge - A Beginner's Guide To btrfs
Seravi - Using RAID with btrfs and recovering from broken disks
John Ramsden - Using Btrfs for Easy Backup and Rollback
HowToForge - Samba Server installation on Ubuntu 16.04 LTS
Eggplant - Faster Samba (SMB / CIFS) Share Performance