Secondary navigation

Running Python on HPC

Establish a graphical connection to the HPC

  1. Connect to the CQU HPC through using graphical connection, how to do so can be found here. Connect to either the Marie or Curie login node.

    To launch python editor

  2. Launch the ‘GNOME Terminal’ located on your desktop
  3. Next you have to load the python module which you wish to use. A list of the possible interpreter software that is available on the CQU HPC can be found here.
  4. As an example we are going to be launching ‘Spyder’ from the anaconda module; to do so run the following commands:
    1. First check for any currently loaded python modules using:
      $which python
      /usr/bin/python
    2. Import the module you wish to use:
      module load anaconda-5.0.1
    3. Check the module loaded correctly:

      which python
      /apps/software/anaconda/5.0.1/bin/python
    4. Select the editor within anaconda in which you wish to use:
      $spyder

Load python code form computer

  1. Coping and opening a python code on the HPC
    1. Create a shared drive following the steps found here.
    2. Copy your python files from your normal computer to a location on the HPC, the desktop will work fine.
    3. In Spyder or other python editor press Ctrl + O to open file, find your file and select open

Searching for a python module installed on our various HPC python software stacks

As some of you might not be aware, we have many different python software stacks installed on our HPC system, with differing python modules installed within them.

To make things easier, I have created a script that you can use to search within all of these python software stacks to see if a python module has been installed and what version it is currently running.

To use this script, simply issue the command

check_python_module.sh <software module>

For example, I get a lot of questions in regards to tensorflow, so you can run the following command to see if this module is installed within all the python software stacks.  So I am search for the module that has the name tensor within it.

bash-4.2$ check_python_module.sh tensor
/apps/modules/modulefiles/anaconda
/apps/modules/modulefiles/anaconda-2.3.0
/apps/modules/modulefiles/anaconda-2.3.0-qiime
/apps/modules/modulefiles/anaconda-2.3.0-qiime2
/apps/modules/modulefiles/anaconda-2.3.0-test
/apps/modules/modulefiles/anaconda2-4.2.0
/apps/modules/modulefiles/anaconda-4.2.0
/apps/modules/modulefiles/anaconda-4.2.0-test
/apps/modules/modulefiles/anaconda-5.0.1
tensorboard                        1.6.0
tensorflow                         1.5.0
tensorflow-tensorboard             1.5.1
/apps/modules/modulefiles/anaconda-python2-5.0.1
tensorboard                        1.6.0
tensorflow                         1.6.0
tensorflow-gpu                     1.6.0
tensorflow-tensorboard             1.5.1
/apps/modules/modulefiles/anaconda-python2-5.0.1-test
/apps/modules/modulefiles/anaconda-python3-2019-10
/apps/modules/modulefiles/anaconda-python3-2020-02
tensorboard                        2.3.0
tensorboard-plugin-wit             1.7.0
tensorflow                         2.3.1
tensorflow-estimator               2.3.0
tensorflow-gpu                     2.1.0
/apps/modules/modulefiles/anaconda-python3-2020-02-test
/apps/modules/modulefiles/deep-learning-packages
tensorboard (1.6.0)
tensorflow-gpu (1.6.0)
/apps/modules/modulefiles/deep-learning-packages2
tensorboard                        1.8.0
tensorflow                         1.8.0
tensorflow-gpu                     1.8.0
/apps/modules/modulefiles/python
tensorboard                        1.8.0
tensorflow                         1.8.0
tensorflow-gpu                     1.8.0
/apps/modules/modulefiles/python2
tensorboard                        1.8.0
tensorflow                         1.8.0
tensorflow-gpu                     1.8.0
/apps/modules/modulefiles/python-2.6.2
tensorboard                        1.8.0
tensorflow                         1.8.0
tensorflow-gpu                     1.8.0
/apps/modules/modulefiles/python-2.7.14
tensorboard                        1.8.0
tensorflow                         1.8.0
tensorflow-gpu                     1.8.0
/apps/modules/modulefiles/python-2.7.15
tensorboard                        1.8.0
tensorflow                         1.8.0
tensorflow-gpu                     1.8.0
/apps/modules/modulefiles/python-2.7.3
tensorboard                        1.8.0
tensorflow                         1.8.0
tensorflow-gpu                     1.8.0
/apps/modules/modulefiles/python-2.7.9
/apps/modules/modulefiles/python3
/apps/modules/modulefiles/python-3.4.3
/apps/modules/modulefiles/python-3.4.9
/apps/modules/modulefiles/python-3.5.9
/apps/modules/modulefiles/python-3.7.4
/apps/modules/modulefiles/python-3.7.6
/apps/modules/modulefiles/python-3.8.0
/apps/modules/modulefiles/python-3.8.2-deep-learning
tensorboard                   2.2.1
tensorboard-plugin-wit        1.6.0.post3
tensorflow-estimator          2.2.0
tensorflow-gpu                2.2.0rc3
bash-4.2$

Thus, for the example above, you can see within the software module “python-3.8.2-deep-learning” it has the following packages installed.

tensorboard                   2.2.1
tensorboard-plugin-wit        1.6.0.post3
tensorflow-estimator          2.2.0
tensorflow-gpu                2.2.0rc3

Please note that if the python module tensorflow-gpu is available, it highlights that version of tensorflow has been built to use GPU’S.

Thus to use this python software stack, all you would need to do is load the appropriate software module.

module load python-3.8.2-deep-learning

Hopefully this little script will make it easier to determine what python stack has the modules installed you require.  If none of them has the module you are chasing, or the version, please do hesitate to ask people via teams or email.