Archiving Data

Users with accounts on mhpcc.hpc.mil systems will be able to archive their data. The archive is mounted on interactive/login nodes typically as /mnt/archive/$LOGNAME.

The ARCHIVE_HOME environment variable points to the actual archive directory.

The archive file system is NOT a high performance file system. Data that is stored on the archive file system is automatically migrated to tape. Accessing data that has been migrated to tape requires a tape mount and copy from tape to disk cache. If the data on the disk cache is not being accessed (file not open) that data is a candidate to be migrated back to tape.

Data on the archive is purged when the user's account is deleted.

All users are encouraged to back up their data to the archive and to use the archive for staging large data sets to/from scratch directories on high performance file systems.

Hints for best use of the archive: Instead of writing hundreds of individual files to the archive, bundle them into one tar file and write the tar file to the archive. This will greatly speed up the retrieval process from the archive. It is faster to pull one tar file from the archive than hundreds of individual files. When retrieving files from the archive, do it file by file in sequential order. Do not pull files from the archive in parallel. The files are on tape which is a sequential access medium.

You may pull/push data from the archive via a script file from any interactive
node. Make sure that the script is excutable e.g chmod +x archive.script. It is suggested that you nohup this script and place it in the background e.g nohup archive.script &

Standard UNIX file/directory commands will function on your archive directory, e.g., ls, mkdir, cp, rmdir, rm.

The pstoolkit archive command is available on the MHPCC Mana system for users with the default login shell of bash or Borne.

To use the pstoolkit to stage files from the ARCHIVE, run a batch job and write resulting output to the ARCHIVE is a process that uses 2 files and 1 command.  A script file to execute the archive “get” command and submit the second file, your command/submit file.  The command is “nohup” to eliminate manual intervention.

A generic script file example:
==========
 #!/bin/bash
cd $WORKDIR/dir (where you want to put the archive data for the run)
archive get -C /mnt/archive/user_id/dir/ file_names
cd $WORKDIR/dir (where command file resides)
qsub submit_file.cmd

A generic submit_file.cmd example:
==========
#!/bin/sh
#PBS -l walltime=00:10:00
#PBS -l select=1:mpiprocs=8:ncpus=8
#PBS -l place=scatter
#PBS -A VWXYZ00000000
#PBS -o file.out
#PBS -e file.error
#PBS -q standard

your setup

mpirun your_executable

your cleanup

/mnt/cfs/pkgs/pstoolkit/bin/archive put -C /mnt/archive/user_id/dir/ file_names

The nohup command:
=========
nohup ./SCRIPT_NAME > out$$ 2>&1 &

Explanation of the above:
“nohup” will cause SCRIPT_NAME to continue to execute after you log off
“out$$” the process ID will be appended to the “out” file, thus making it unique
“2>&1” standard out/error will be redirected to the file out$$ (will contain PBS job id)
“&” puts everything in the background so you can continue to use the terminal session

ps -p PID (to check on the progress of the SCRIPT_NAME)

For users that regularly perform this function, the command is:

archive get -C archive_directory_structure file_name
archive put -C archive_directory_structure file_name

Concisely,
- create a script that gets your archive data and submits a command file
- add the archive put in your command file
- execute the nohup command

Transfer Queue Available

============
#!/bin/bash
#PBS -l walltime=00:##:00
#PBS -l select=1:mpiprocs=1:ncpus=1
#PBS -l place=scatter
#PBS -o transfer.o
#PBS -e transfer.e
#PBS -A ABCDE00000000
#PBS -q transfer

TRANSFER_DIR=$ARCHIVE_HOME/transfer/$PBS_JOBID

if [ ! -d $TRANSFER_DIR ]; then mkdir -p $TRANSFER_DIR || exit 1;
fi

echo "starting transfer"

cd $WORKDIR

tar -pzcvf $TRANSFER_DIR/file_name.tgz file_name
============

This is an efficient and effective tool for archiving data and we encourage you to use the archive queue routinely. Please contact the CCAC or MHPCC Help Desk if you require assistance.

Limits

As 500GB is the size of one tape cartridge. it is not advisable due to performance and increased risk of data loss, to have files larger than 500GB on the archive system, as spanning tapes is not recommended. Large/Huge files take time to be written to tape, consequently they reside on the disk cache until they are completely written to tape and then released from the disk cache. This may cause an issue with your transfer and the archive system as a whole.

It is advised that all files be less than 500 GB to reduce the probability of any unforeseen issues from occurring.

EXAMPLE: tar -zcvlO <files|dirs> | split -b 5120m - $ARCHIVE_HOME/mylargefile.tgz

For more information please visit our Mana documentation.

Hints for best use of the archive:

Instead of writing many individual files to the archive, bundle them into one tar file and write the tar file to the archive. This will greatly speed up the retrieval process from the archive. It is faster to pull one tar file from the archive than many individual files. Please keep the size of your tar file under 500GB. When retrieving files from the archive, do it file by file in sequential order. Do not pull files from the archive in parallel. The files are on tape which is a sequential access medium.

Transferring Archive Data to a Remote Site

If you have many small files on the archive (see Hints for best use of the archive) bundle them into a tar file of no larger than 500GB. It can take over 4 days to transfer a 500GB file to a remote location.

nohup tar -pzcf $WORKDIR/MY_FILE.tar.tgz $ARCHIVE_HOME/MY_DIR/ &

To transfer a file from the archive to a remote location:

kinit
nohup scp $ARCHIVE_HOME/file_name user@host:remote_file &