wiki:DataSharing

DataSharing

Data can be shared via uploading to or downloading from one of our sftp servers.

Getting an account

To connect to one of the sftp servers, you will need to generate a public/private key pair, request an account and have your public key linked to that account.
For instructions see: Request an account

What is stored where

A description of the general layout of the storage systems with personal, group and public folders as well as their status is documented in Storage SOP - What is stored where.

SFTP servers

We no longer ship data on hard drives, but as homage to good ol' snail mail we named the SFTP servers for the research environment after famous homing pigions.

User Community Server name Key fingerprints (in various formats) Named after
UMCG Research cher-ami.hpc.rug.nl SHA256: 7yoZzaCYBRT8oCYaHUg3ACsQZ/M98irw5XyXkMtKH6Q=
MD5: 8e:61:15:9c:f4:1d:12:f7:04:a3:3a:8e:c0:2d:dd:df
Cher Ami
Genome Diagnostics zf-ds.gcc.rug.nl SHA256: AkenUdD5DSe2nz+WDPZkfIC4ZKzuORHFulT9wjnpipM=
MD5: 62:3a:f8:2b:e3:c2:8f:65:22:d0:92:5e:7f:f5:13:aa
---
Genome Diagnostics lz-ds.gcc.rug.nl SHA256: ohpSL7XngrgXr7qwB3JwV4TqtYG5ojG7q27aHnkIdGI=
MD5: 31:e5:72:03:08:c5:db:0f:29:97:e6:b4:b7:28:a1:86
---

Data Transfer via the Secure File Transfer Protocol (SFTP)

  1. Using a graphical user interface on Linux, Unix and Mac OS X
  2. Using a graphical user interface on Windows
  3. Using the command line

1. Via SFTP using a graphical user interface on Linux, Unix and Mac OS X

Step 1: Download and install FileZilla

Download and install FileZilla: http://filezilla-project.org/download.php?type=client

Step 2: Launch the application and disable the remote directory tree view

Please disable the remote directory tree

Step 3: Open the Site Manager

Step 3: Specify connection details and save those as a preset

  • Click on New Site
    • Host: Get a servername from the list
    • Port: leave empty
    • Protocol: SFTP - SSH File Transfer Protocol
    • Logon Type: Normal
    • User: youraccountname
    • Password: leave empty
  • Click Connect to save your new site preset and open a connection.

Step 4: Check the server's digital fingerprint

If this is the first time you connect to the sftp server, FileZilla will:

  • Show you the digital fingerprint of the server's key.
  • Ask if it is correct.
  • Ask if you want to trust the server and save the key in FileZilla's cache.

Check if the fingerprint matches the one for the server you selected from the list and:

  • if it's a match: click Ok to continue.
  • if the fingerprint is different: do not continue and contact us a.s.a.p.!
  • if this Unknown host key dialog does not pop-up and you cannot connect: see Troubleshooting below.

Step 5: Enter the password for your private key

  • You may be prompted for the password of your private key.
  • On Mac OS X the OS may have stored the password for your private key in the Keychain Access application. If this is the case and you previously allowed the ssh application to access the password for your private key stored in your Keychain, you will not be prompted to type your private key's password (again).
  • If FileZilla does not prompt for the password of your private key and you cannot connect: see Troubleshooting below.

Step 6: Transfer data

The system should establish a connection and FileZilla will present you with a file browser similar to the one shown in the screenshot below.

  • Your local files are shown on the left and files on sftp server are shown on right.
  • If you receive an Error: Authentication failed message instead and cannot connect: see Troubleshooting below.

Troubleshooting

Problem: I do not get the 'Unknown host key' dialog or I am not prompted for the password of my private key and cannot login.
Solution: Sometimes FileZilla will not handle the first connection properly. By connecting for the first time via SSH in a Terminal, you can add the server to the list of known hosts and the password to the keychain, which FilleZilla will then happily use for any following connection. Please try this:

  1. Quit FileZilla
  2. Launch the Terminal application (/Applications/Utilities/Terminal.app)
  3. A terminal window should open
  4. Type ssh youraccountname@servername.hpc.rug.nl
  5. The system will show a warning about the "unknown host key" if this is the first time you are connecting.
    For example for cher-ami.hpc.rug.nl you should get:
    The authenticity of host 'cher-ami.hpc.rug.nl (129.125.60.103)' can't be established.
    RSA key fingerprint is SHA256:7yoZzaCYBRT8oCYaHUg3ACsQZ/M98irw5XyXkMtKH6Q.
    Are you sure you want to continue connecting (yes/no)?
    
    If the fingerprint is the same as above: Type yes to accept the key. If the fingerprint is different: Do not accept the key. Some other server is pretending to be the server you try to connect to. Contact us a.s.a.p.
  6. The system will ask for the password of your private key. Provide the password and check the checkbox to save the password in your keychain.
  7. You should be able to get in and the server will tell you something like this:
    Warning: Permanently added 'cher-ami.hpc.rug.nl,129.125.60.103' (RSA) to the list of known hosts.
    
    If you have an sftp-only account you may get an error that ssh access is disabled: that is Ok.
  8. Logout by typing exit and closing the Terminal application.
  9. Re-try with FileZilla

Problem: I am prompted for the password of my private key, but cannot login.
Solution: If you have multiple keys and or keys stored in non-default locations, you may be trying to connect with the wrong key. By default OpenSSH, which provides the encryption for the S in SFTP, will look for a private key in ~/.ssh/id_rsa or ~/.ssh/id_dsa for keys generated with the RSA or DSA encryption algorithms, respectively. The ~ is an alias for your home directory. If you stored your key file in a different location either move the key to the default location or check the OpenSSH and FileZilla documentation on how to use a key from a non-default location.

2. Via SFTP using a graphical user interface on Windows

Step 1: Download and install WinSCP

Download and install WinSCP: http://winscp.net/eng/download.php

(NOTE: we suggest not to use FileZilla on Windows as it removes the password from private keys when importing them. This is a potential security leak.)

Step 2: Launch WinSCP

Step 3: Specify connection details and save those as a preset

  • File protocol: SFTP (The Port number will now change to 22, which is the default for sftp.)
  • Host name: Get a servername from the list
  • User name: youraccountname
  • Password: leave empty
  • Private key file: select your private key in *.ppk format (which you generated with PuTTYgen).

Click Save... to store these connection details for later use and click Login to connect.

Step 4: Check the server's digital fingerprint

If this is the first time you connect to the sftp server, WinSCP will:

  • Show you the server's digital fingerprint.
  • Ask if it is correct.
  • Ask if you want to save it.

Check if the fingerprint matches the one for the server you selected from the list and:

  • if it's a match: click Yes to continue.
  • if the fingerprint is different: do not continue and contact us a.s.a.p.!

Step 5: Enter the password for your private key

Step 6: Transfer data

The system should establish a connection and WinSCP will present you with a file browser similar to the one shown in the screenshot below. Your local files are shown on the left and files on sftp server are shown on right.

3. Via SFTP on the commandline

You can use the standard ftp commandline tool, but this one can be a hassle to use as it does not support tab completion of path names nor automatically creates folders when you want to copy data recursively. You may want to try lftp instead if you have it installed. On our HPC cluster User Interface (UI) servers lftp is available. For lftp:

Step 1: start interactive lftp shell

$> lftp

You will now get an lftp prompt as indicated by lftp :~> in the steps below. Your prompt may be formatted differently.

Step 2: configure SSH

If your private key is in the default location (~/.ssh/id_rsa for RSA based keys or ~/.ssh/id_dsa for DSA based keys) you can skip this step. When your private key is not stored in the default location you need to tell lftp to use ssh with the -i option to specify the path to your private key file.

lftp :~> set sftp:connect-program "ssh -a -x -i /path/to/your/private_key_file"

Step 3: open a connection to the SFTP server

Replace youraccountname with the name of your account and keep the ,none suffix to tell lftp we are not using a password based login. Note you have to specify both the port number 22 with -p and make sure the server address is prefixed with sftp://

lftp :~> open -u youraccountname,none -p 22 sftp://server.name.nl

Step 4: transfer data

Use the get and put commands to download or upload data from the sftp server to cluster, respectively.

download data from sftp to cluster

get FILE.file

upload data from cluster to sftp

put FILE.file

Lftp does not support the -r switch with get and put for recursive data transfers, but it provides the mirror command instead. By default mirror will download directories. To upload with the mirror command you'll need to use mirror -R for reverse mirror. Hence to download to the current dir (where you started lftp):

lftp :~> mirror folder_on_remote_server

To upload to the current destination_folder_on_remote_server:

lftp :~> cd destination_folder_on_remote_server
lftp :~> mirror -R folder_on_local_server
  • If you receive an Fatal error: Host key verification failed message instead and cannot execute any FTP commands: see Troubleshooting below.

Step 5: logout once you are done

lftp :~> exit

Troubleshooting

Problem: I cannot execute any FTP commands and get a "host key verification" error.
Solution: Your commandline client may not handle the first connection properly. By connecting for the first time via SSH in a Terminal, you can add the server to the list of known hosts. Please try this:

  1. Quit your commandline sftp client
  2. Launch a Terminal application (On Mac OS X systems this is located in /Applications/Utilities/Terminal.app)
  3. A terminal window should open
  4. Type ssh youraccountname@sftp.server.name.nl
  5. The system will show a warning about the "unknown host key" if this is the first time you are connecting.
    For example for cher-ami.hpc.rug.nl:
    The authenticity of host 'cher-ami.hpc.rug.nl (129.125.60.103)' can't be established.
    RSA key fingerprint is SHA256:7yoZzaCYBRT8oCYaHUg3ACsQZ/M98irw5XyXkMtKH6Q.
    Are you sure you want to continue connecting (yes/no)?
    
    If the fingerprint is the same as above: Type yes to accept the key. If the fingerprint is different: Do not accept the key. Some other server is pretending to be the server you try to connect to. Contact us a.s.a.p.
  6. The system will ask for the password of your private key. Provide the password and optionally save the password in a password manager (called 'Keychain' on Mac OS X).
  7. You should be able to get in and the server will tell you something like this:
    Warning: Permanently added 'cher-ami.hpc.rug.nl,129.125.60.103' (RSA) to the list of known hosts.
    
    If you have an sftp-only account you may get an error that ssh access is disabled: that is Ok.
  8. Logout by typing exit and closing the Terminal application.
  9. Re-try with your commandline sftp client
Last modified 7 months ago Last modified on 2017-03-21T12:52:40+01:00

Attachments (12)

Download all attachments as: .zip