PLEASE NOTE: ALL DATA ON THE DATA DISKS IS DELETED AFTER TWO WEEKS
BACK UP YOUR DATA IMMEDIATELY AFTER YOUR RUN!
SCP
FROM THE ALS, data may be scp’ed from your user account by typing:
scp -pr datadirectory username@destination:/datadirectory
where destination is the name of your home institution computer.
EXAMPLE: scp -pr /data/bl502/arnold ArnoldS@hollywood.edu:/arnold/awesomedata
NOTE: The -r is for recursive, so that a directory and all its contents are copied, while the -p preserves the original date on the file. At home, you can then type ls -Flart *.img to list the image files in the order in which they were taken. If you want to copy a single file or several files, use:
scp -p filename username@destination:/datadirectory (or *.img in place of the filename)
To SCP FROM OFFSITE, log onto the download machine by typing one of the following commands:
ssh username@bcsb-cnx.als.lbl.gov
You will be logged into your home directory so you can scp processed data directly. For raw images you must go to the directory that corresponds to the beamline at which you collected your data. For example, type cd /data/bl822/yourdata. Once in the correct directory you can scp as indicated above.
Alternatively, you can pull data data in from your remote location with scp:
scp -r username@bcsb-cnx.als.lbl.gov:/data/bl???/directory/where/the/data/is .
(don’t forget the “.” at the end of the command – this indicates to put the data in the current directory, where you ran the scp command)
FTP
First, tar your image files using the command tar -cvf name.tar directory where directory is the directory full of processed data.
- At the prompt, type ftp, or sftp sitename (i.e. ftp home.colorado.edu)
- At the ftp prompt type passive
- At the ftp prompt type prompt
- At the ftp prompt type bin
- At the ftp prompt type cd datadirectory (where datadirectory is the directory at the destination server where you want to store your files)
- At the ftp prompt type put name.tar
- When done, type bye to end the session
RSYNC
To back up all the data from a source directory into a destination directory on your drive, use:
rsync -ravz source_directory destination_dir
If you are logging in from offsite, you can log in through rsync. For instance:
rsync -ravz user@bcsb-cnx.als.lbl.gov:/data/etc/ destination_dir
where /data/etc is your directory with processed data, and destination is the directory you want the data to be stored in.
HARD DRIVE (from onsite)
Plug in the firewire or usb cable to the linux box. Firewire cables are provided at most data processing machines. Plug in the power supply (if necessary) and switch the drive on. Wait 5-20 seconds. A window with the /firewire directory should appear. Drag and drop your files. When done, right click the disk drive icon on the desktop and select “unmount volume”, or type “umount /firewire” in the prompt. Be sure to unmount before unplugging to prevent loss of data.
If the drive is Mac OS formatted (HFS+), Linux may not recognize the drive. If no processing machine recognizes the drive, you may have to reformat it.
Notes about External Hard Drive Issues
Not all external hard drives will work with Linux after they are plugged in. The first issue is that the external hard drive may be the wrong format. When formatting a drive using Windows or a Mac, the default filesystem on the drive will be NTFS or HFS+, and neither of those filesystems can be written to by Linux. The second issue is that the ownership of the drive may be wrong. This is because a Linux filesystem stores the file’s owner as a user ID number, and not the name. Since the user ID numbers are different between the BCSB network and other organizations, you will likely encounter “permission denied” errors.
Please note that the drive must be assigned the device file /dev/sdb. If, for any reason, the drive is assigned a different device file (sdc, etc…), you will need to reboot the machine so that Linux can free up and reset the device file assignment sequence. The reason for this problem is other users forgetting to properly unmount their drives before unplugging them. Please remember to always unmount drives before unplugging.
Changing ownership of the drive
To change ownership of the drive, mount the drive, and use the command:
sudo /usr/local/bin/changeowner [your username here]
This command recursively runs through the file tree of the external hard drive and reassigns the user to the one that you specify.
Reformatting a drive
Sometimes it’s better to just wipe a drive and use that space. To do so, run the script:
/usr/local/bin/formathd
It should take you through the steps necessary to reformat the drive. In the process, it will ask you to unplug the drive and reattach it. This is necessary as a workaround for not having ‘root’ privileges.
The drive will be reformatted using the ext3 filesystem, and will not be compatible with Windows or Mac OS X machines. In later versions of this utility, there will be options for formats.
If you have any further questions, please contact our software team.
Thank you for reading!