Sushi101

Summary

Sushi is a small cluster shared by several Stack@CS faculty members.

Sushi Nodes

There are 10 sushi nodes. Each node has:

• 48 cores
• 256GB RAM
• Local scratch space -- a few hundred GB in /tmp
• Shared access to a shared /home file system -- 180TB over NFS
• 10Gig Ethernet connection to the other nodes
• Access to the external Internet

Sushi Access

Only certain labs have access to Sushi. If you are not sure, ask the lab PI.

If you have an account on sushi, you can access sushi via the head node: sushi.cs.vt.edu (128.173.236.117 on the intranet)

• scp external files to your home directory on the head node
• Launch jobs from the head node

Sushi Jobs

Launch jobs from the head node using the PBS job submission system. There are many guides to PBS on the web. I recommend Purdue's guide.

DO NOT run expensive tasks on the head node itself. This affects the cluster's stability and inconveniences everyone.

Learning to use sushi

Before you use sushi for the first time, you should:

• Read this wiki page
• Learn about the PBS system
• Review the man page for qsub, qstat, and qnodes
• Try a simple practice job, e.g. an "echo" that prints the node name

This may take you a day or two. It is well worth the investment.

Example

Here's what I do for "embarassingly parallel" jobs driven by an input file with one task per line.

Split input into files, one task per line

 (10:51:16) davisjam@sushi-headnode ~/qsub-jobs/Memo/input $ split sl-regex-filteredForPrototype-all.json sl-regex-filteredForPrototype-all-piece-  --lines=3000 --additional-suffix=.json --numeric-suffixes --suffix-length=4

Write job script

I use this as a template and tweak it from there. You might also try the GNU Parallel tool. There's a copy in /home/davisjam/bin/parallel.

 (12:05:29) davisjam@sushi-headnode ~/qsub-jobs/Memo $ cat qsub-memo.sh
 #!/usr/bin/env bash
 
 # You must provide REGEX_FILE
 # e.g. "qsub -v REGEX_FILE='/home/davisjam/qsub-jobs/RegexRepl/syntax/input/test/500.json' qsub-syntax.sh"
 
 #########################################
 ## PBS Configuration (Single Comment # ONLY)
 #########################################
 #
 #PBS -l nodes=1:ppn=8
 #
 # Save all env vars -- including PERL5LIB
 #PBS -V
 #########################################
 
 #########################################
 ## Setup
 #########################################
 
 #OUT_FILE=~/data/syntax/cross-registry-real/`basename $REGEX_FILE .json`-slras-job$PBS_JOBID.json
 OUT_FILE=~/data/memo/all-SL/`basename $REGEX_FILE .json`-measureMemo-job$PBS_JOBID.pkl.bz2
 
 STDOUT_FILE=$HOME/logs/qsub-memo-$$.out
 STDERR_FILE=$HOME/logs/qsub-memo-$$.err
 
 NCORES=`wc -l < $PBS_NODEFILE`
 
 # Flush NFS?
 rm $STDOUT_FILE 2>/dev/null
 rm $STDERR_FILE 2>/dev/null
 sync; sync; sync; sync; sync;
 touch $STDOUT_FILE
 touch $STDERR_FILE
 
 # Here we go!
 echo "Hello on node " `hostname` " with $NCORES cores"
 echo "REGEX_FILE $REGEX_FILE"
 echo "OUT_FILE $OUT_FILE"
 echo "STDOUT_FILE $STDOUT_FILE STDERR_FILE $STDERR_FILE"
 
 export MEMOIZATION_PROJECT_ROOT=~/memoized-regex-engine
 export ECOSYSTEM_REGEXP_PROJECT_ROOT=~/EcosystemRegexps
 
 set -x
 
 # For data about prototype
 # PYTHONUNBUFFERED=1 $MEMOIZATION_PROJECT_ROOT/eval/measure-memoization-behavior.py \
 #   --regex-file $REGEX_FILE \
 #   --queryPrototype \
 #   --trials 1 \
 #   --queryProductionEngines \
 #   --parallelism $NCORES \
 #   --out-file $OUT_FILE \
 #   > $STDOUT_FILE \
 #   2>$STDERR_FILE
 
 # For data about other regex engines -- use if you want to test with extended features not supported by prototype
 PYTHONUNBUFFERED=1 $MEMOIZATION_PROJECT_ROOT/eval/measure-memoization-behavior.py \
   --regex-file $REGEX_FILE \
   --useCSharpToFindMostEI \
   --queryProductionEngines \
   --parallelism $NCORES \
   --out-file $OUT_FILE \
   > $STDOUT_FILE \
   2>$STDERR_FILE

Launch job

 (10:56:30) davisjam@sushi-headnode ~/qsub-jobs/Memo $ for f in input/500-piece-*; do echo $f; qsub -v REGEX_FILE=`pwd`/input/$f qsub-memo.sh; done

Monitor job

 (11:13:54) davisjam@sushi-headnode ~/qsub-jobs/Memo $ ls -lhtra ~/logs

(and tail log files, etc.)

Export data

If you want to export the data (e.g. for analysis in a Jupyter notebook), try something like this:

 (11:15:32) davisjam@sushi-headnode ~/qsub-jobs/Memo $ mkdir ~/export-latest; cp ~/data/memo/all-SL/*.pkl.bz2 ~/export-latest; tar -czvf ~/export-latest.tgz ~/export-latest; scp ...

Handy scripts

Check on your jobs

How much longer will you be waiting?

summarize-job-state.pl

 #!/usr/bin/env perl
 # Author: Jamie Davis <davisjam@vt.edu>
 # Description: Summarize the status of the jobs of a user
 
 use strict;
 use warnings;
 
 if (scalar(@ARGV) ne 1) {
   die "  Summarize state of jobs submitted by a user\nusage: $0 username\n";
 }
 my $user = $ARGV[0];
 if (length($user) < 1) {
   die "Error, username is empty\n";
 }
 
 my @lines = `qstat -u $user`;
 my @running = grep { m/\s+$user\s+.*\sR\s/ } @lines;
 my @queued = grep { m/\s+$user\s+.*\sQ\s/ } @lines;
 my @error = grep { m/\s+$user\s+.*\sE\s/ } @lines;
 
 my $nRunning = scalar(@running);
 my $nQueued = scalar(@queued);
 my $nError = scalar(@error);
 my $nJobs = $nRunning + $nQueued + $nError;
 
 print "    Running jobs: $nRunning\n";
 print "    Queued jobs: $nQueued\n";
 print "    Error jobs: $nError\n";
 print " + ------------------------\n";
 print "    Active jobs: $nJobs\n";

Abort a run

Sometimes you see an error show up in your logs files and need to abort the run.

kill-my-jobs.pl

 #!/usr/bin/env perl
 # Author: Jamie Davis <davisjam@vt.edu>
 # Description: Kill (qdel) all jobs owned by the given user
 
 use strict;
 use warnings;
 
 if (scalar(@ARGV) ne 1) {
   die "  qdel all jobs submitted by a user\nusage: $0 username\n";
 }
 my $user = $ARGV[0];
 if (length($user) < 1) {
   die "Error, username is empty\n";
 }
 
 my @jobIDs = &getJobIDs($user);
 
 if (@jobIDs) {
   &log("qdel'ing the " . scalar(@jobIDs) . " jobs owned by $user");
   my $cmd = "qdel " . join(" ", @jobIDs);
   system($cmd);
 } else {
   print "qstat reported no jobs to kill\n";
 }
 
 ###########
 
 sub getJobIDs {
   my ($user) = @_;
 
   &log("Using qstat to get the jobs owned by $user");
   my @qstat_output = `qstat -u $user`;
   chomp @qstat_output;
   my @jobLines = grep { m/\s+$user\s+/ } @qstat_output;
   my @jobIDs = map {
     my ($id) = ( $_ =~ m/^(\d+)\./ );
     $id;
   } @jobLines;
   return @jobIDs;
 }
 
 sub log {
   my ($msg) = @_;
   print STDERR "$msg\n";
 }

Clean up /tmp

Sometimes my analysis tools leak files into /tmp on the sushi nodes.

clean-my-tmp.pl

 #!/usr/bin/env perl
 # Author: Jamie Davis <davisjam@vt.edu>
 # Description: Print commands to clean up my files in /tmp across sushi
 
 use strict;
 use warnings;
 
 my @nodes = qw/ sushi01 sushi02 sushi03 sushi04 sushi05 sushi06 sushi07 sushi08 sushi09 sushi10 /;
 
 ## Parse args
 if (scalar(@ARGV) < 1 or scalar(@ARGV) > 2) {
   die "Print commands to delete files in /tmp on each sushi node [matching the specified find predicates]
 Usage: $0 owning-user ['find predicates']
 
 Examples:
   $0 davisjam
     - Deletes all files owned by davisjam in /tmp on all sushi nodes
   $0 davisjam '-name \"protoRegexEngine*\"'
     - Delete all files ... whose name matches this predicate
       You should wrap predicates in single-quotes, and use double-quotes for any quoting within the predicates
       (The ssh command is wrapped in single-quotes)
 ";
 }
 my $user = $ARGV[0];
 if (length($user) < 1) {
   die "Error, username is empty\n";
 }
 my $findPredicates = "";
 if (scalar(@ARGV) >= 2) {
   $findPredicates = $ARGV[1];
 }
 
 ## Cleanup operations
 my $cmd = "ssh $nodes[0] 'find /tmp -user $user -delete -type f $findPredicates'";
 
 for my $node (@nodes) {
   my $cmd = "find /tmp -user $user -delete -type f $findPredicates";
   print("ssh $node '$cmd' &\n");
 }
 
 &log("\n^^ If the preceding commands are to your liking, copy/paste/execute to run them.");
 
 ############
 
 sub log {
   my ($msg) = @_;
   print STDERR "$msg\n";
 }