test guide HTML Intensive computing with R Intensive computing with R I ve included further information and included some questions for open discussion. I hope you find this useful. General information R is available on all servers, packages can be installed on compute64d (CHECK others!). General information about high performance computing (eg you need to be connected via VPN if you login remotely): See https://info.maths.ed.ac.uk/itsupport/hpc.html (RUBEN+NATALIA: Is VPN needed? I always connect without VPN on xserver2) Connecting to servers The School of Mathematics has several gateway servers which you can use to connect to the Maths network from external locations, either logging in to an SSH terminal session or starting a remote desktop session. VPN is not required through this route. For example, you first connect via ssh to one gateway server ssh UUN@xserver2.maths.ed.ac.uk and then, you connect via ssh again to a compute server, for example ssh UUN@compute64d.maths.ed.ac.uk If your local computer does not have any ssh tools installed, you can login to the Maths SSH servers via the web, see SSH via a web browser. Note that you must have Java installed on the local machine. Further information can be found at https://info.maths.ed.ac.uk/itsupport/hpc/maths-compute-servers.html (RUBEN+NATALIA: Is VPN needed? I always connect without VPN on xserver2) (RUBEN+NATALIA: Not sure if we need /usr/bin/ssh? It think it would also be worth adding a short section on .bashrc for aliases, further config, etc.) (IOANNIS: I have made some changes to the above. Can you both check you can connect to a compute server through xserver2?) Recommended editors: We advise using either Rstudio or Emacs ESS. Both editors support code indentation, can be used to connect to a bash terminal and further, include Git support. In Rstudio, command lines can be launched from scripts using Ctrl+Alt+Enter (Windows, Linux) or Cmd+Alt+Enter (MACOS), so you do not need to copy and paste. In Emacs The maths computing servers (eg compute64d) run on GNU/Linux operating system servers. Instructions how to work with the terminal and the unix scripting language are available here: Getting started: https://info.maths.ed.ac.uk/itsupport/faq/linux-faq/linux-getting-started.html Further: https://info.maths.ed.ac.uk/itsupport/faq/linux-faq/linux-commands.html Once you log in, you are on your H drive. (RUBEN+NATALIA: Would it not be better to have the below instructions more generic in a separate section? For example under a section "Basics of filesystem" or otherwise) The commands I use to move to subfolder “Folder” of the current folder are cd Folder Command ls lists the subfolders and files in the current folder. Once you log in, change the directory to where you want to save files, run R by typing “R” in the command line. Run your scripts as usual. Quit R by typing q() TODO Basics of filesystem State "TODO" from [2019-10-05 Sat 11:20] TODO Transferring files State "TODO" from [2019-10-05 Sat 11:20] Enabling graphical interface X11 forwarding (-X) To enable graphical interface so that you can plot images interactively when working with a terminal under a remote connection you would need to forward X11, i.e., ssh -X UUN@xserver2.maths.ed.ac.uk and then establish a secure shell connection with a compute server ssh UUN@compute64d.maths.ed.ac.uk If X11 is not forwarded, then calls to plotting functions in R such as the plot function will automatically save the image in the working directory (getwd()). (RUBEN+NATALIA: Do we need the following here?) Command save.image saves everything, or use command save to save a list of variables. On the client side, the -X (capital X) option to ssh enables X11 forwarding, and you can make this the default (for all connections or for a specific conection) with ForwardX11 yes in ~/.ssh/config. Example of ~/.ssh/config file Host compute64d HostName compute64d.maths.ed.ac.uk Port 22 User UUN ServerAliveInterval 100 ProxyCommand ssh xserver2 nc %h %p ForwardAgent yes ForwardX11 yes On the server side, X11Forwarding yes must specified in /etc/ssh/sshd_config. Note that the default is no forwarding (some distributions turn it on in their default /etc/ssh/sshd_config), and that the user cannot override this setting. Thankfully, this is not the case in maths compute servers. Emacs Tramp mode and helm-tramp * TRAMP is a package for editing remote files. TRAMP uses a remote shell connection (ssh) and it can transfer the files using rcp or a similar program, or it can encode the file contents and transfer them right through the shell connection. It allows you transparent access to files on remote access. “Transparent” means that usually the user doesn’t have to worry about anything. All that changes is the filename convention to indicate that the file resides on a remote system. One of the neat things about using Tramp via a remote shell, is that Emacs will then automatically invoke other remote shell commands directly on that server, e.g. when viewing a remote file, try M-x shell, M-x rgrep, etc. To enable the graphical interface when you connect on maths server with TRAMP, you may find the prescribed settings in ~/.ssh/config in the previous section helpful. (IOANNIS: Check with Steve) Restoring idle-sessions. By default, if you close the terminal, any process that you are running will be interrupted. To avoid this, you can use tmux, a terminal multiplexer (allows a user to access multiple separate terminal sessions inside a single terminal window or remote terminal session). tmux enables a number of terminals to be created, accessed, and controlled from a single screen. tmux may be detached from a screen and continue running in the background, then later reattached. When tmux is started it creates a new session with a single window and displays it on screen. A status line at the bottom of the screen shows information on the current session and is used to enter interactive commands. A session is a single collection of pseudo terminals under the management of tmux. Each session has one or more windows linked to it. A window occupies the entire screen and may be split into rectangular panes, each of which is a separate pseudo terminal. Any number of tmux instances may connect to the same session, and any number of windows may be present in the same session. Once all sessions are killed, tmux exits. Each session is persistent and will survive accidental disconnection (such as ssh(1) connection timeout) or intentional detaching (with the C-b d key strokes). tmux may be reattached using: tmux attach For more information, see man tmux. Remember that only two processes per server are allowed (Question: is the latter sentence Is this correct? Does it refer to tmux or any processes on the server?) Further information and training unix If you want to understand how the unix terminal works and acquire skills in shell programming (a scripting language) we recommend Unix 1, Unix 2 and Unix 3 courses. Additionally, these Linux introductions may be useful Introduction to Linux from the Linux Documentation Project Introduction to UNIX from the Imperial College git - the stupid content tracker: Git is a fast, scalable, distributed revision control system with an unusually rich command set that provides both high-level operations and full access to internals. It is designed for coordinating work among programmers, but it can be used to track changes in any set of files. Its goals include speed, data integrity, and support for distributed, non-linear workflows. Some useful resources: Happy Git and GitHub for the useR Quick Introduction to Version Control with Git and GitHub guide: a minimal tutorial 10 Git Tutorials for Beginners Immersion Cheat Sheet Some other useful references Some examples to get started ## Connecting to the server /usr/bin/ssh UUN@compute64a.maths.ed.ac.uk /usr/bin/ssh UUN@compute64d.maths.ed.ac.uk /usr/bin/ssh UUN@compute64e.maths.ed.ac.uk /usr/bin/ssh UUN@compute64e2.maths.ed.ac.uk /usr/bin/ssh UUN@compute64h.maths.ed.ac.uk (/usr/bin/ssh is distracting) ## Connecting to the server tmux # creating a terminal session R # launching R inside the created terminal session tmux ls # list of sessions tmux attach -t 0 # reattach to session 0 tmux attach -t 1 # reattach to session 1 tmux kill-session -t 0 # kill session 0 # Ctrl+B tmux commands (advanced use, I never use it) # % split vertically # " split horizontally # arrows move around panes # z maximize/minimize # c new window # x kill window # number move among windows # d detach # tmux command list: # https://alvinalexander.com/linux-unix/tmux-cheat-sheet-commands-pdf TODO Installing R packages in your local directory /home/username/R (compute servers) State "TODO" from [2019-10-06 Sun 09:48] IN-PROGRESS Temporary installation of JAGS (To be replaced soon) State "IN-PROGRESS" from [2019-10-04 Fri 18:10] IT ticket issued but Steve suggested 2 week delay on tmux. Probably for JAGS too. JAGS is available on compute64a, compute64d, compute64e, compute64e2 and compute64h. A JAGS installation exists in a shared folder and by default, R cannot find JAGS. To use JAGS. It is reported that there is a "compiler problem with R" in the server and so you have to use R installed locally in the shared folder. (RUBEN: What is meant by "compiler problem". Can you investigate what the problem is exactly?) After connecting to server, run the following line once, then the shared folder will be included in you ‘bashprofile’ file. (No need to run next connection) cat /home/s1532523/shared/append_to_profile >> ~/.bash_profile The command above appends the following to your ~/.bash_profile SHARED="/home/s1532523/shared" export PATH="$SHARED/bin:$PATH" export LD_LIBRARY_PATH="$SHARED/lib:$LD_LIBRARY_PATH" export PKG_CONFIG_PATH="$SHARED/lib/pkgconfig:$PKG_CONFIG_PATH" Solution due to s1532523. (RUBEN+NATALIA: Can we write this solution in a better way?) Before running R, source bash_profile to make sure that you use R in the shared folder. source ~/.bash_profile Run R and check which R you are using via R.home() function. Local : "/home/s1532523/shared/lib64/R" Server : "/usr/lib64/R" (RUBEN+NATALIA: Is the solution unclear? Do you think we can improve it?) If you are using locally installed R, you can install “rjags” package. REFILE PhD Tutorial State "REFILE" from [2019-10-05 Sat 12:01] What are your thoughts on this? Key PhD people to include? This article was published on 2025-04-22
HTML Intensive computing with R Intensive computing with R I ve included further information and included some questions for open discussion. I hope you find this useful. General information R is available on all servers, packages can be installed on compute64d (CHECK others!). General information about high performance computing (eg you need to be connected via VPN if you login remotely): See https://info.maths.ed.ac.uk/itsupport/hpc.html (RUBEN+NATALIA: Is VPN needed? I always connect without VPN on xserver2) Connecting to servers The School of Mathematics has several gateway servers which you can use to connect to the Maths network from external locations, either logging in to an SSH terminal session or starting a remote desktop session. VPN is not required through this route. For example, you first connect via ssh to one gateway server ssh UUN@xserver2.maths.ed.ac.uk and then, you connect via ssh again to a compute server, for example ssh UUN@compute64d.maths.ed.ac.uk If your local computer does not have any ssh tools installed, you can login to the Maths SSH servers via the web, see SSH via a web browser. Note that you must have Java installed on the local machine. Further information can be found at https://info.maths.ed.ac.uk/itsupport/hpc/maths-compute-servers.html (RUBEN+NATALIA: Is VPN needed? I always connect without VPN on xserver2) (RUBEN+NATALIA: Not sure if we need /usr/bin/ssh? It think it would also be worth adding a short section on .bashrc for aliases, further config, etc.) (IOANNIS: I have made some changes to the above. Can you both check you can connect to a compute server through xserver2?) Recommended editors: We advise using either Rstudio or Emacs ESS. Both editors support code indentation, can be used to connect to a bash terminal and further, include Git support. In Rstudio, command lines can be launched from scripts using Ctrl+Alt+Enter (Windows, Linux) or Cmd+Alt+Enter (MACOS), so you do not need to copy and paste. In Emacs The maths computing servers (eg compute64d) run on GNU/Linux operating system servers. Instructions how to work with the terminal and the unix scripting language are available here: Getting started: https://info.maths.ed.ac.uk/itsupport/faq/linux-faq/linux-getting-started.html Further: https://info.maths.ed.ac.uk/itsupport/faq/linux-faq/linux-commands.html Once you log in, you are on your H drive. (RUBEN+NATALIA: Would it not be better to have the below instructions more generic in a separate section? For example under a section "Basics of filesystem" or otherwise) The commands I use to move to subfolder “Folder” of the current folder are cd Folder Command ls lists the subfolders and files in the current folder. Once you log in, change the directory to where you want to save files, run R by typing “R” in the command line. Run your scripts as usual. Quit R by typing q() TODO Basics of filesystem State "TODO" from [2019-10-05 Sat 11:20] TODO Transferring files State "TODO" from [2019-10-05 Sat 11:20] Enabling graphical interface X11 forwarding (-X) To enable graphical interface so that you can plot images interactively when working with a terminal under a remote connection you would need to forward X11, i.e., ssh -X UUN@xserver2.maths.ed.ac.uk and then establish a secure shell connection with a compute server ssh UUN@compute64d.maths.ed.ac.uk If X11 is not forwarded, then calls to plotting functions in R such as the plot function will automatically save the image in the working directory (getwd()). (RUBEN+NATALIA: Do we need the following here?) Command save.image saves everything, or use command save to save a list of variables. On the client side, the -X (capital X) option to ssh enables X11 forwarding, and you can make this the default (for all connections or for a specific conection) with ForwardX11 yes in ~/.ssh/config. Example of ~/.ssh/config file Host compute64d HostName compute64d.maths.ed.ac.uk Port 22 User UUN ServerAliveInterval 100 ProxyCommand ssh xserver2 nc %h %p ForwardAgent yes ForwardX11 yes On the server side, X11Forwarding yes must specified in /etc/ssh/sshd_config. Note that the default is no forwarding (some distributions turn it on in their default /etc/ssh/sshd_config), and that the user cannot override this setting. Thankfully, this is not the case in maths compute servers. Emacs Tramp mode and helm-tramp * TRAMP is a package for editing remote files. TRAMP uses a remote shell connection (ssh) and it can transfer the files using rcp or a similar program, or it can encode the file contents and transfer them right through the shell connection. It allows you transparent access to files on remote access. “Transparent” means that usually the user doesn’t have to worry about anything. All that changes is the filename convention to indicate that the file resides on a remote system. One of the neat things about using Tramp via a remote shell, is that Emacs will then automatically invoke other remote shell commands directly on that server, e.g. when viewing a remote file, try M-x shell, M-x rgrep, etc. To enable the graphical interface when you connect on maths server with TRAMP, you may find the prescribed settings in ~/.ssh/config in the previous section helpful. (IOANNIS: Check with Steve) Restoring idle-sessions. By default, if you close the terminal, any process that you are running will be interrupted. To avoid this, you can use tmux, a terminal multiplexer (allows a user to access multiple separate terminal sessions inside a single terminal window or remote terminal session). tmux enables a number of terminals to be created, accessed, and controlled from a single screen. tmux may be detached from a screen and continue running in the background, then later reattached. When tmux is started it creates a new session with a single window and displays it on screen. A status line at the bottom of the screen shows information on the current session and is used to enter interactive commands. A session is a single collection of pseudo terminals under the management of tmux. Each session has one or more windows linked to it. A window occupies the entire screen and may be split into rectangular panes, each of which is a separate pseudo terminal. Any number of tmux instances may connect to the same session, and any number of windows may be present in the same session. Once all sessions are killed, tmux exits. Each session is persistent and will survive accidental disconnection (such as ssh(1) connection timeout) or intentional detaching (with the C-b d key strokes). tmux may be reattached using: tmux attach For more information, see man tmux. Remember that only two processes per server are allowed (Question: is the latter sentence Is this correct? Does it refer to tmux or any processes on the server?) Further information and training unix If you want to understand how the unix terminal works and acquire skills in shell programming (a scripting language) we recommend Unix 1, Unix 2 and Unix 3 courses. Additionally, these Linux introductions may be useful Introduction to Linux from the Linux Documentation Project Introduction to UNIX from the Imperial College git - the stupid content tracker: Git is a fast, scalable, distributed revision control system with an unusually rich command set that provides both high-level operations and full access to internals. It is designed for coordinating work among programmers, but it can be used to track changes in any set of files. Its goals include speed, data integrity, and support for distributed, non-linear workflows. Some useful resources: Happy Git and GitHub for the useR Quick Introduction to Version Control with Git and GitHub guide: a minimal tutorial 10 Git Tutorials for Beginners Immersion Cheat Sheet Some other useful references Some examples to get started ## Connecting to the server /usr/bin/ssh UUN@compute64a.maths.ed.ac.uk /usr/bin/ssh UUN@compute64d.maths.ed.ac.uk /usr/bin/ssh UUN@compute64e.maths.ed.ac.uk /usr/bin/ssh UUN@compute64e2.maths.ed.ac.uk /usr/bin/ssh UUN@compute64h.maths.ed.ac.uk (/usr/bin/ssh is distracting) ## Connecting to the server tmux # creating a terminal session R # launching R inside the created terminal session tmux ls # list of sessions tmux attach -t 0 # reattach to session 0 tmux attach -t 1 # reattach to session 1 tmux kill-session -t 0 # kill session 0 # Ctrl+B tmux commands (advanced use, I never use it) # % split vertically # " split horizontally # arrows move around panes # z maximize/minimize # c new window # x kill window # number move among windows # d detach # tmux command list: # https://alvinalexander.com/linux-unix/tmux-cheat-sheet-commands-pdf TODO Installing R packages in your local directory /home/username/R (compute servers) State "TODO" from [2019-10-06 Sun 09:48] IN-PROGRESS Temporary installation of JAGS (To be replaced soon) State "IN-PROGRESS" from [2019-10-04 Fri 18:10] IT ticket issued but Steve suggested 2 week delay on tmux. Probably for JAGS too. JAGS is available on compute64a, compute64d, compute64e, compute64e2 and compute64h. A JAGS installation exists in a shared folder and by default, R cannot find JAGS. To use JAGS. It is reported that there is a "compiler problem with R" in the server and so you have to use R installed locally in the shared folder. (RUBEN: What is meant by "compiler problem". Can you investigate what the problem is exactly?) After connecting to server, run the following line once, then the shared folder will be included in you ‘bashprofile’ file. (No need to run next connection) cat /home/s1532523/shared/append_to_profile >> ~/.bash_profile The command above appends the following to your ~/.bash_profile SHARED="/home/s1532523/shared" export PATH="$SHARED/bin:$PATH" export LD_LIBRARY_PATH="$SHARED/lib:$LD_LIBRARY_PATH" export PKG_CONFIG_PATH="$SHARED/lib/pkgconfig:$PKG_CONFIG_PATH" Solution due to s1532523. (RUBEN+NATALIA: Can we write this solution in a better way?) Before running R, source bash_profile to make sure that you use R in the shared folder. source ~/.bash_profile Run R and check which R you are using via R.home() function. Local : "/home/s1532523/shared/lib64/R" Server : "/usr/lib64/R" (RUBEN+NATALIA: Is the solution unclear? Do you think we can improve it?) If you are using locally installed R, you can install “rjags” package. REFILE PhD Tutorial State "REFILE" from [2019-10-05 Sat 12:01] What are your thoughts on this? Key PhD people to include?
