Lattice Gauge Theory at Brookhaven National Laboratory (BNL)

The Layout File provides a mapping between QCDOC nodes and PFS directories. Its is unique to each partition and mapping. The PFS directory is what each node uses for IO via NFS (in QCDOC, every Compute Node is capable of IO). Such mappings are necessary when splitting, scattering, gathering and unsplitting Lattice Configuration or Propagator files.
The first line in a Layout File (see a qlayout file example) contains an interger representing the number of machine dimensions, while the second line provides the number of nodes in each dimension. What follows then is a number of lines equal to the number of nodes in the machine partition with each line mapping the QMP node number to the corresponding PFS directory.

To generate a Layout File you will need to source a qcsh script located at $CRE_HOME/bin/qlayout.qcsh. This script may take a single argument specifying the filename of the output layout file. Rememebr, since the Layout File is unique to the axis mapping of a partition, you should first map the axis before generating the layout file.
The following example is for a single MBd partition (64 nodes):

(qcdochostb:/home/user:QCSH)% qpartition_remap -T015 -X2 -Y3 -Z4 
(qcdochostb:/home/user:QCSH)% source $CRE_HOME/bin/qlayout.qcsh layout.X2.Y3.Z4.T015

NOTE: in PBS, the Layout File is automatically generated when the user plans to QSPLIT or QUNSPLIT a file. You must use our version 5 of the PBS scripts for this feature to work. Your $QBATCH_HOME env. variable sould be set to: /qcdoc/local/batch/v5. For more info: Qbatch File Management .

NOTE: Dru Renner/ Carleton DeTar have wriiten wrapper scripts that generate Layout files based on the generated exit.info.* files of a job. The users first need to copy the exit files into a single directory (call it exits) and then run the make_layout.sh script passing as arguments the exits directory and then the sizes in each dimension. The make_layout.sh script has been copied into the $CRE_HOME/bin directory (03/01/2007).
Here is an example creating a layout file. After the job has finished and the exit.info.* files have been generated:

mkdir exits
cp /somewhere/exit.info.* exits/.
make_layout.sh exits 2 8 2 2 
cp layout.exits layout.X2.Y3.Z4.T015

qsplit

Description:

Both utilities qsplit and qsplitDWF are wrappers around QIO executables that split QIO files into part files. The QIO executables are, of course, host (single CPU) compliled binaries located at $HOST_QIO_HOME/bin/.

Usage:

*qsplit*	<File> <Destination_Directory> <Layout_File>
*qsplitDWF*	<File> <X> <Y> <Z> <T> [<S>]

File: The filename of the file to be split. Also, the file prefix of the part files that are generated.
Destination_Directory:: The directory on the front-end host where the part files are placed.
Layout File: The Layout file that is used for the splitting.
<X> <Y> <Z> <T> [<S>]: sizes in each dimension. Used only with qsplitDWF

Example:

qsplit qio_file.out ./test/ myLayout.out

qsplitDWF qio_file.out 2 2 2 2 2

qscatter

Description:

qscatter scatters part files placed in a single directory (Source_directory) by qsplit to the appropriate PFS directories.
The files that will be scattered have the same prefix in their filenames. This prefix is indicated by the argument File. The general format of the part file is File.vol.NNNN.

Usage:

qscatter <Source_Directory> <PFS_destination_Directory> <File> <Layout_File>

Source_Directory:: The directory on the front-end host where the part files are located.
PFS_Destination_Directory:: The directory onto the PFS system where part files will be placed.
For example, if set to data a part could be placed in: /pfs/acc7/ACC07/B1/M2/D31/A1/data/
File: The file prefix of the files to be scattered.
Layout File: The Layout file that was used for the splitting.

Example:

qscatter ./temp/ ./ qio_file.out qlayout.out

NOTE: The qsplit-ing and qscatter-ing can be easily done in one step using QIO executables:
$HOST_QIO_HOME/bin/qio-convert-mesh-pfs 0 SingleFile < Layout_File

qgather

Description:

qgather gathers all part files from PFS directories into a local directory on the front-end. qgather does not put the files together into a single file; this is the job of qunsplit.
The files that are being gathered have the same prefix in their filenames. This prefix is indicated by the argument File. The general format of the part file is File.vol.NNNN.

Usage:

qgather <PFS_Source_Directory> <Destination_Directory> <File> <Layout_File>

PFS_Source_Directory: The PFS directory where the part files are located. For example, if set to data one of the part files could be placed in: /pfs/acc7/ACC07/B1/M2/D31/A1/data/.
Destination_Directory: The directory onto the front-end where all part files will be gathered.
File: The file prefix of the files to be gathered.
Layout File: The Layout file

Example:

qgather data/ ./ qio_file.out qlayout

NOTE: Dru/Carleton have a similar utility (shell script) called cppfs that is now (03/01/2007) copied to $CRE_HOME/bin/ so it should be in everyone's $PATH. The arguments to cppfs are similar to those of qscatter:

 cppfs LayoutFile File DestinationDirectory

qunsplit

Description:

qunsplit puts together all part files located in a directory (Source_Directory) into a single file. There are three versions on qunsplit: qunsplitILDG, qunsplitSCIDAC and qunsplitDWF. qunsplit simply calls qunsplitSCIDAC.
The part files that are to be put together should all have the same prefix indicated by the argument File. The general format of the part file is File.vol.NNNN
The argument File also specifies the filename of the final product, the merged file.

Usage:

*qunsplit*	<Source_Directory> <File> <Layout_File>
*qunsplitILDG*	<Source_Directory> <File> <Layout_File>
*qunsplitSCIDAC*	<Source_Directory> <File> <Layout_File>
*qunsplitDWF*	<File> <X> <Y> <Z> <T> [<S>]

Source_Directory: The directory where the part files are located.
File: The file prefix of the files to be merged and also the output filename.
Layout File: The Layout file
<X> <Y> <Z> <T> [<S>]: sizes in each dimension. Used only with qunsplitDWF

Other Utilities

cppfs LayoutFile File DestinationDirectory: copies all files indicated by File from the PFS directories listed in the LayoutFile to the DestinationDirectory on the front-end host (by Dru/Carleton)
Similar to qgather above.
lspfs LayoutFile File: List all files indicated by File in the PFS directories in the LayoutFile
rmpfs LayoutFile File: Delete all files indicated by File in the PFS directories in the LayoutFile