====== Running Jupyter Notebooks and JupyterLab Remotely ====== This is a summary of the setup to run [[https://jupyter-notebook.readthedocs.io/en/stable/index.html|Jupyter notebooks]] or [[https://jupyterlab.readthedocs.io/en/stable/index.html|JupyterLab]] on a remote server. Steps 1-4 describe the configuration of the setup and only have to be done once. Step 5 describes the usage of the setup. ===== 1. SSH Configuration ===== Generally, if you want to log into a remote server (e.g., [[https://tikoraluk.mathstat.dal.ca/wiki/doku.php|tikoraluk]]) using the ''ssh'' command, you would use the following command in your terminal: ssh @ where '''' and '''' are your user name on and domain (e.g., ''tikoraluk.mathstat.dal.ca'') of the //remote// machine. Typing in your user name and the full domain of your server can be cumbersome, but you can set up a configuration file that allows you to log in using an alias. On your //local// computer, open a terminal, and update the file ''config'' in the folder ''.ssh'' in your home directory: touch ~/.ssh/config If the file did not yet exist, this will create an empty file for you. Open this file with a text editor of your choice (note that the folder ''~/.ssh'' might be hidden) and add the following lines at the end of that file: Host tiko HostName tikoraluk.mathstat.dal.ca User Here, ''tiko'' is the alias that you can choose freely, ''tikoraluk.mathstat.dal.ca'' is the domain of the remote server, and '''' is your user name on that machine. Save the file with the name ''config'' in the ''.ssh'' folder in your local home directory. Now you should be able to log in to the remote server using the following command in your terminal: ssh tiko You will be prompted to enter your password on the the remote machine. To log out of the remote server use the command ''ctrl+d'' or type ''exit''. ===== 2. Passwordless SSH ===== If you do not want to enter your password every time, you can set up an ssh-key that gives you passwordless access to the remote server. Open an terminal on your *local* machine and check if you already have an existing key pair by listing the contents of the ''.ssh'' folder in you home directory: ls -l ~/.ssh You are looking for the two files ''id_rsa'' and ''id_rsa.pub''. If these files do not exist, you can generate them with the following command in your terminal: ssh-keygen -b 4096 The ''-b 4096'' option specifies the byte size of the key. You will be prompted to enter a passphrase, but you can leave it empty to skip this extra level of protection. If the key generation was successful, you will be shown a randomart image in your terminal. Copy the public key to the remote server: ssh-copy-id tiko Note that this assumes that you have configured your ssh connection following the instructions above. If you did not set up an alias you will have to replace ''tiko'' by ''@tikoraluk.mathstat.dal.ca'', where '''' is your user name on the remote server. Test the passwordless connection by logging in to the remote machine: ssh tiko If you chose a passphrase for your key, you will be prompted to enter it the first time. Afterwards, you should be able to connect to the remote server without it. To log out of the remote server use the command ''ctrl+d'' or type ''exit''. ===== 3. Conda Environment ===== To keep your code portable and reproducible, it is recommended to create a separate [[https://docs.conda.io/projects/conda/en/latest/user-guide/concepts/environments.html|conda environment]] for each of your projects. This will also protect you from breaking your code if you have to update your Python packages in the future. The best way to create and curate your conda environment is using an ''environment.yml'' file that lists all the Python packages that you need for your project. In the following, we assume that you are logged in and working on the remote machine, but the same principles apply to the work on your local computer. Let us also assume that you have a folder ''my_project'' that contains everything related to your project. We will refer to this as the project folder. Typically, this folder would contain sub-directories that separate your data, code, literature, and write-ups with a structure similar to this: my_project/ ├── data/ │   ├── external/ │   ├── processed/ │   └── raw/ ├── notebooks/ ├── README.md ├── references/ ├── reports/ │   └── figures/ └── src/ (You can [[https://github.com/christophrenkl/create_project|download]] a simple shell program, that creates a similar folder structure for you.) Open a new file with a text editor of your choice and add the following name: my_project # name of your environment channels: # channels to download packages from - conda-forge - defaults dependencies: # list of packages you want to install - python=3.9 # here we choose Python version 3.9 - cartopy - cmocean - jupyter # make sure to add the jupyter and - jupyterlab # jupyterlab packages - matplotlib - numpy - pandas - xarray (Add or remove packages depending on your needs.) Save the file as ''environment.yml'' in your project folder. To create or [[https://docs.conda.io/projects/conda/en/latest/user-guide/tasks/manage-environments.html?highlight=--prune#updating-an-environment|update your conda environment]], (e.g., if you had to add another package), open a terminal and log in to the remote server. Change into the project folder and install the packages for the environment. cd ~/projects/my_project/ conda env update --file environment.yml --prune You only have to do this once or every time you want to install a new package. Once your conda environment is created, you need to activate it: conda activate my_project You will have to run this command every time you open a new terminal or log in to the remote server to run jupyter. ===== 4. Jupyter Configuration Files ===== By default, the [[https://jupyter-notebook.readthedocs.io/en/stable/security.html|Jupyter notebook server is protected]] by a token that you have to enter every time you run it on a remote machine or if you run it with the ''--no-browser'' option. (When you run Jupyter on your local machine this is directly parsed into the URL if the notebook server opens your browser automatically). Alternatively, you can set up a [[https://jupyter-notebook.readthedocs.io/en/stable/config.html|configuration file]] to enable password authentication which is generally more convenient to work with. Check if you already have a configuration file. Assuming that you are still logged in to the remote server and have a conda environment activated that includes the Jupyter packages, list the contents of the ''.jupyter'' folder in your home directory: ls -l ~/.jupyter You are looking for the files ''jupyter_notebook_config.py'' and `''jupyter_lab_config.py'' which are the configuration files for instances of Jupyter notebook and Jupyter lab, respectively. If these files do not exist, you can create them using the commands jupyter notebook --generate-config jupyter lab --generate-config To set the password for Jupyter notebooks, run the command jupyter notebook password This will ask you to enter and verify a password of your choice. Similarly, to set the password for Jupyter labs, run the command jupyter lab password This will also ask you to enter and verify a password of your choice and can be the same as for Jupyter notebooks. You can also use these commands to reset or change your password. There are a variety of command line arguments that can be used to to run a notebook server. Please refer to the [[https://jupyter-notebook.readthedocs.io/en/stable/config.html|documentation]] for the available options. ===== 5. Open Remote Jupyter Notebook or Jupyter Lab ===== On your local computer, open a terminal and log in to the remote server: ssh tiko After successful login, change into the project directory and activate the conda environment: cd ~/projects/my_project/ conda activate my_project If you have issues with ''conda activate'', try ''source activate my_project''. Open Jupyter lab with the following options jupyter lab --no-browser --port=XXXX Change ''XXXX'' to the port number of your choice, typically 8889. It is possible that this port is occupied. In that case, try 8887, 8890, 8787, or similar. This will start a Jupyter lab on the remote server and since you specified the ''--no-browser'' option, you will only see some status messages in the terminal. Open a second terminal on your local computer and log in to the remote server, but this time using the port forwarding option ''-L'' and binding the remote port ''XXXX'' specified above to a local port: ssh -N -f -L localhost:8888:localhost:XXXX tiko Finally, open the URL [[http://localhost:8888/]] in a browser of your choice. You will be prompted to enter your password specified in [[#4. Jupyter Configuration Files|Step 4]] above. Once you are finished with your work, close the browser and stop the remote Jupyter notebook by using the ''ctrl+c'' command twice in the first terminal. To log out of the remote server use the command ''ctrl+d'' or type ''exit''. In the second terminal, stop the port forwarding with lsof -ti:8888 | xargs kill -9 ==== 5.1. Port Forwarding in Configuration File ==== You can make your life easier by creating and alias in the [[[#1. SSH Configuration|SSH configuration file]]. Open the file ''config'' in the hidden folder ''.ssh'' in your home directory with a text editor of your choice and add the following lines at the end: Host tiko-jupyter HostName tikoraluk.mathstat.dal.ca User LocalForward 8888 localhost:XXXX Replace '''' by your user name and ''XXXX'' by the port number that you have found to be available above. Instead of typing the lengthy command in the second shell, you can now use ssh -N -f tiko-jupyter