Lattice Gauge Theory at Brookhaven National Laboratory (BNL)

To access BNL QCDOC computing resources (including copying files to/from QCDOC servers) you must use the QCDOC ssh gateways. One way (2-hop) to transfer files is to first copy them into the gateways and then transfer them to the appropriate QCDOC file system.

If the remote site does not use a gateway, files may be transferred to BNL in one hop if you execute the scp command on the BNL host. For example, you can transfer files from Columbia (qcdochost4.phys.Columbia.edu) to the US QCDOC front-end node if you run the scp command on the qcdochostb host:

qcdochostb:scp user@qcdochost4.phys.columbia.edu:/home/user/ker* .

You can avoid typing passwords/passphrases if you have your ssh keys set up. Instructions on how to do this can be found here.

ssh Tunneling

Another way to transfer files from remote hosts is via ssh tunneling (port forwarding). This way you can transfer files directly to the QCDOC file systems (in one hop) without first copying them onto the ssh gateways. An ssh tunnel can be setup manually or you can use an ssh tunneling script as described below.

NOTE: If you are using a windows machine, you can still create an ssh tunnel to the QCDOC front-end nodes. Detailed instructions can be found here.

Creating an ssh tunnel manually

There are many instructions on the web on how to set up ssh port forwarding. Here is an example: To set up an ssh tunnel from a remote (outside BNL) host, such as qcdochost4.phys.columbia.edu, to the US DOE front-end node qcdochostb.qcdoc.bnl.gov do:

qcdochost4:>ssh -N -f -L 2123:qcdochostb.qcdoc.bnl.gov:22 user@ssh.qcdoc.bnl.gov

[BNL security Banner removed to conserve space]

user@ssh.qcdoc.bnl.gov's password:

You can choose any port on your local host as long as it is not a privileged port (here 2123, should be greater than 1023). To transfer files to qcdochostb.qcdoc.bnl.gov via the tunnel (notice the capital P in the scp command):

qcdochost4:>scp -P 2123 kerbuf.* user@localhost:/home/user/.

[BNL security Banner removed to conserve space]

user@qcdochostb's password:

You can set qcdcohostb as an alias to localhost by editing the /etc/hosts file on the remote host (you can have more than one aliases for localhost). For example, on qcdochost4.phys.columbia.edu the first line in /etc/hosts is:

127.0.0.1 loopback localhost qcdochostb

This way you can use qcdochostb in place of localhost in the above ssh command:

qcdochost4:>scp -P 2123 kerbuf.* user@qcdochostb:/home/user/.

NOTE:The ssh tunnel will remain open until you close it. When you are done with the file transfers, find the process ID of the ssh tunnel and kill it.

Using the tunnel script

Instead of opening the tunnel manually, you can use a Perl script (by Steve Adler) that can:

Create a tunnel
ssh/scp via the tunnel (tssh, tscp)
Close the tunnel when done (tclose).

The web site with instructions about the tunnel script is at: here

The above script has been installed on the QCDOC front-end node.

Installing the tunnel script

Download and untar the script on the machine from which you access BNL or JLab. Run the ./install.pl script to install it. If you are not a root user, put the binary in your own area:

qcdochost4:> ./install.pl
Enter the directory where you want tunnel installed [/usr/local/bin]: /home/user/tunnel

Every user needs to create a config file at $HOME/.tunnel/config on the remote host. In this file you will put a line such as:

qcdochost4:> more .tunnel/config
qcdochostb.qcdoc.bnl.gov:2010:ssh.qcdoc.bnl.gov:user:qcdochostb:bnl

NOTE 1: You may have several lines in your config file, one per remote host.

NOTE 2: The fields in the config line: qcdochostb.qcdoc.bnl.gov is the destination host at BNL (qcdi01.jlab.org at JLAB), 2010 is the local port that will be used by the tunnel (you can use any port greater than 1024), ssh.qcdoc.bnl.gov is the ssh gateway at BNL (one at JLAB is login.jlab.org), user is the username, qcdochostb is an alias to localhost (in /etc/hosts), and the last field is unique abbreviation of you choosing.

NOTE 3: If localhost has not being aliased to qcdochostb in /etc/hosts, then in the fifth field on the above line you should use localhost, instead of qcdochostb.

NOTE 4: ssh records RSA host public keys for the hosts you connect to in .ssh/known_hosts or .ssh2/hostkeys. If localhost has not being aliased in /etc/hosts, the destination host is known to ssh as localhost. If you already have an RSA host key for localhost in the above files, ssh may not allow you to go through the tunnel because it thinks the host identification has been changed. In this case, you should remove or rename the known_hosts or hostkeys entry before running tscp.

Patching the tunnel script

To make the tunnel script compatible with both Linux and AIX, you need to comment-out line 316 on the tunnel.pl script and replace it with the following:

# open(PSFLDS,"ps -efw|");
open(PSFLDS,"ps -e -o user,pid,ppid,ppid,etime,tty,time,args |");

The un-patched script works fine on Linux, but on AIX tclose cannot find the open ssh tunnel and thus doesn't close it.

Using the tunnel script

You can then then use tscp the same way you use scp:

qcdochost4:> tscp kerbuf.* user@bnl:/home/stratos/.
Firing up tunnel to qcdochostb.qcdoc.bnl.gov through ssh.qcdoc.bnl.gov using port 2010
user@ssh.qcdoc.bnl.gov's password:
user@qcdochostb's password:

Once the tunnel has been opened, it will stay open so subsequent tscp commands use the tunnel and require entering your password only once, just as conventional scp.

When you are finished with the tunnel, you should close it with the tclose command. If you don't, the tunnel remains open and the port is tied up.

qcdochost4:> tclose
Port to check is: 2010

Tunnel through port 2010 is active
Closing tunnel on port 2010 going to qcdochostb.qcdoc.bnl.gov through ssh.qcdoc.bnl.gov

Example: Transferring files from FNAL to BNL or JLAB

To transfer files from FNAL to BNL or JLAB you should tunnel (either manually or using the tunnel script) through the BNL ssh gateways or the JLAB gateways. Here is an example of the .tunnel/config script:

lqcd.fnal.gov:> more .tunnel/config
qcdochostb.qcdoc.bnl.gov:2010:ssh.qcdoc.bnl.gov:user:localhost:qcdochostb qcdi01.jlab.org:2271:login.jlab.org:user:localhost:qcdi01

The fields in the config file are explained above. You can then use tscp to push or pull files from the remote sites:

lqcd.fnal.gov:> tscp user@qcdochostb:notes .

[BNL banner removed]

Password:
user@localhost's password:

You will be asked for your gateway password first and then for your password on the remote host. When you are done using the tunnel, remember to close it:

lqcd.fnal.gov:> tclose

Transferring files to FNAL using kerberized clients

Kerberos clients have been installed on the USDOE front-end host. To transfer files to FNAL you first need to obtain a kerberos ticket (using kinit) and then use rcp or fscp to transfer files from/to FNAL:

qcdochostb:>kinit
Password for user@FNAL.GOV:

qcdochostb:>klist
Ticket cache: FILE:/var/krb5/security/creds/krb5cc_pid
Default principal: user@FNAL.GOV

Valid starting Expires Service principal
05/20/05 15:38:16 05/21/05 15:38:08 krbtgt/FNAL.GOV@FNAL.GOV

qcdochostb:>rcp testFile user@lqcd.fnal.gov:.

qcdochostb:>rcp user@lqcd.fnal.gov:testFile testFile.bak

ssh and scp on qcdochostb are not kerberized. Instead, Don Holmgren has patched ssh with the FNAL patches and installed the kerberized versions fssh and fscp. SO, after kinit, you can use fscp to transfer files to FNAL.

If you use a non-kerberized utility (ssh on qcdochostb is not kerberized) you will be prompted for your password. If you hit enter at the password prompt, you will be prompted with a cryptocard challenge:

qcdochostb:>ssh lqcd.fnal.gov
user@lqcd.fnal.gov's password:

Red Hat Linux release 7.1 (Seawolf)
Kernel 2.4.25 on a 4-processor i686

Press ENTER and compare this challenge to the one on your display: [22569055]
Enter the displayed response:

For more on using FNAL kerberos, see: http://lqcd.fnal.gov/kerberos_notes.html

The kerberos 5 package can be obtained from NCSA: http://www.ncsa.uiuc.edu/UserInfo/Consulting/Security/Software.html