HowTo:CS Stash

Introduction

CS Stash is a shared departmental storage system designed for sharing files to Linux based servers and workstations. The service is based on Ceph, and more specifically CephFS: https://docs.ceph.com/en/reef/cephfs/ The goal of the service is to provide fault tolerant, secure, high-speed file sharing to Linux machines in the department. This project is currently in it's pilot state, so space is very limited right now.

Creating an Allocation

All allocations are created by Techstaff. Contact Techstaff to make your request for storage space.

Installing Ceph Client

To mount your allocation, you will need to install the ceph client, if it is not already installed. You will need root level access to install ceph. See: https://docs.ceph.com/en/latest/install/ for more details and the latest information on installing ceph. Here are some brief instructions, using the cephadm tool that should work on any supported Linux system (Redhat, Rocky, CentOS, Ubuntu, Debian, Alma, Etc.).

• Download the cephadm tool
  • curl --silent --remote-name --location https://download.ceph.com/rpm-reef/el9/noarch/cephadm
• Make the tool executable
  • chmod +x cephadm
• Install the reef ceph repo (Note: Older OSs might have to go down the quincy release, for example Rocky Linux 8)
  • sudo ./cephadm add-repo --release reef
• Install the ceph client
  • sudo ./cephadm install ceph-common

Mounting Your Allocation

When your allocation is created, you will be given the following information needed to mount your filesystem:

• <username>
• <secret_key>
• <path>

You need to save the contents of the <secret_key> into a file, and secure the file. For example, in the file /etc/.<username>.secret with chmod mode 0600.

You can have the filesystem automatically mounted on boot by adding an entry to your /etc/fstab file, substitute your information as needed. You can optionally mount a sub-path of your top level path.

• <username>@.cephfs=<path>[/<sub-path] <mount_location> ceph mon_addr=stash.cs.vt.edu:3300,secretfile=<path_to_secret_file>,_netdev,noatime,wsync,rbytes,ms_mode=secure 0 0

You may get notices about files in your /etc/ceph directory, but they can be ignored.

Security

• Your <secret_key> should be kept secret, anyone with access to the <username> and <secret_key> can mount, and have full access to your filesystem
• The CephFS mount works like a locally mounted POSIX filesystem with ACLs enabled. You can change file ownership and permissions like you would a local filesystem.
• All data is stored encrypted at rest
• If you use the "ms_mode=secure" mount option, then the data will also be encrypted on wire. This is the recommended option.

Quotas

CephFS supports quotas. You can set a specific quota on any individual directory that effects that directory and any sub-directories. For full details, see: https://docs.ceph.com/en/latest/cephfs/quota/ Here is a quick example of setting a 1GB quota on a directory of your mount:

• setfattr -n ceph.quota.max_bytes -v $(numfmt --from=iec 1G) /mnt/ceph/test

Performance

My benchmarks test show that performance from a 1Gbit network connection is about equivalent to a single local spindle drive. Performance from a 10Gbit network connection is much greater than a single local spindle drive.

Backups

Techstaff currently does not do any automatic backups of the data on the CS Stash service.

• The service is fault tolerant to hardware failures
• CephFS offers snapshot support that can help mitigate data loss due to accidental deletion. See: https://docs.ceph.com/en/reef/dev/cephfs-snapshots/ for more details.
• Here is a brief example of creating a snapshot of a subdirectory in your allocation:

user@localhost:/# cd /mnt/ceph/test
user@localhost:/mnt/ceph/test# echo "Version 1" > version.txt
user@localhost:/mnt/ceph/test# cd .snap
user@localhost:/mnt/ceph/test/.snap# mkdir my_snapshot
user@localhost:/mnt/ceph/test/.snap# cd ..
user@localhost:/mnt/ceph/test# echo "Version 2" > version.txt
user@localhost:/mnt/ceph/test# cat version.txt
Version 2
user@localhost:/mnt/ceph/test# cat .snap/my_snapshot/version.txt
Version 1
user@localhost:/mnt/ceph/test#

• Here is a very simple bash script that would keep a one week rolling list of snaphots for a directory

#!/bin/bash

DIR=/mnt/ceph/test
NAME=$(date +%A)

rmdir "$DIR/.snap/$NAME"
mkdir "$DIR/.snap/$NAME"

• If off-site backup is a priority, then an outside solution will need to be implemented