edit · print · PDF

Please note that all the SIEpedia's articles address specific issues or questions raised by IAC users, so they do not attempt to be rigorous or exhaustive, and may or may not be useful or applicable in different or more general contexts.

Step by step guide for the installation and configuration of a cluster to run ParaView

Introduction

As part of a student project that I was mentoring, I built a small ParaView server (4 nodes, 16 CPUs). Since the ParaView documentation didn't seem to have a step-by-step guide on how to do it, I put together the notes of what I did. I'm sure many things could be done better, and I would appreciate any comments on how to improve things. Some of the decisions taken for the installation of our cluster reflect the type of cluster that we wanted to have, and therefore the guide is certainly not comprehensive, but I put it here hoping that it can be useful to someone else. Certainly, some issues (like X connections) seem to be a common source of errors, and the way we solved it can probably be useful to others.

Our test cluster is made up of four x86_64 nodes (each with four CPUs) connected via a Gigabit switch, and the frontend has two network cards. eth1 is connected to the outside world, while eth0 (as per the other nodes) is connected to the switch. In order to make the installation of the cluster easier, I decided to use the Rocks cluster kit, in order to have the basic cluster up and running as quickly as possible (although whether you use other cluster kit or even build the basic cluster manually should not matter much for the installation of ParaView).

Installation of the basic cluster, with Rocks 5.1

The latest version of Rocks is 5.2, but we needed to have the PVFS2 parallel file system installed, and this version does not have (at the time I did the installation) the PVFS2 roll (a pre-packaged and ready-to-install bundle), so we went for Rocks 5.1 instead. The installation of Rocks to create the basic cluster is very easy, and only a few steps are needed.

  • I downloaded the Jumbo DVD x86_64 and the PVFS2 roll from http://www.rocksclusters.org/wordpress/?page_id=97. Basically you put the Jumbo DVD in the CD tray, start the computer and follow the steps (if using a USB DVD drive, you should check out this tip.
  • You can find the Rocks 5.1 documentation here, and first you should install the frontend, following this. For the ParaView cluster, when asked, we select the following rolls to install: base, ganglia, hpc, kernel, os, web-server, and pvfs2 (for this you will need to swap DVDs). This is very similar to a standard distribution install, so you should decide on which partitioning you want (I selected autopartition), network details, keyboard layout, etc.
  • Once the installation of the frontend is finished, when you open for the first time a terminal, you will see that the ssh keys are created automatically. Verify that the network works OK, and before proceeding further, let's make a small change to make sure that the compute nodes will have an X server running (not usual in a computational cluster, but necessary to run ParaView). For this, we follow these instructions.
(:source lang=shell:)
[root@xerenade ~]# cd /export/rocks/install/site-profiles/5.1/nodes/
[root@xerenade nodes]# cp skeleton.xml extend-compute.xml
  • And modify extend-compute.xml so that it becomes:
(:source lang=shell:)
<?xml version="1.0" standalone="no"?>

<kickstart>

  <package type="meta">base-x</package>
  <package type="meta">gnome-desktop</package>
  <package type="meta">x-software-development</package>

  <!-- For the pretty Gnome icons. -->
  <package>indexhtml</package>

  <!-- support packages used to configure X -->
  <package>pygtk2</package>
  <package>pygtk2-libglade</package>


<post>

/sbin/chkconfig xfs on

</post>

</kickstart>
  • And then do:
(:source lang=shell:)
[root@xerenade nodes]# cd /export/rocks/install/
[root@xerenade install]# rocks create distro
  • With this, the compute nodes will have all the necessary stuff to run ParaView, so we can proceed with their installation. For this, we follow these instructions. We run insert-ethers as root in the frontend, and proceed as per the mentioned instructions, although when asked about appliance type, we select Compute-PVFS2 I/O, which means that our nodes will be both computing nodes, and part of the PVFS2 file system (in a production cluster, these roles would probably belong to different nodes, but for our test cluster, this is OK). Installing everything does take a while, and the command rocks-console is useful, but the VNC connections are not always available, so you should also verify the status of the installations with ping, ssh, etc.
  • When the compute nodes have been installed, it is time to configure PVFS2, and for this we just need to do the step mentioned here, which basically is just running rocks sync pvfs. With this in place, we can now verify that PVFS2 works fine:
(:source lang=shell:)
[root@xerenade ~]# touch /mnt/pvfs2/delete.me 
[root@xerenade ~]# rocks run host 'ls -lt /mnt/pvfs2/delete.me'
xerenade: -rw-r--r-- 1 root root 0 Nov 16 22:22 /mnt/pvfs2/delete.me
pvfs2-compute-0-0: -rw-r--r-- 1 root root 0 Nov 16 22:22 /mnt/pvfs2/delete.me
pvfs2-compute-0-1: -rw-r--r-- 1 root root 0 Nov 16 22:22 /mnt/pvfs2/delete.me
pvfs2-compute-0-2: -rw-r--r-- 1 root root 0 Nov 16 22:22 /mnt/pvfs2/delete.me
[root@xerenade ~]#
  • The Rocks 5.1 installation comes with a web server, with Ganglia and some documentation, but the iptables rules as configured by Rocks are too restrictive for our setting. The file /etc/sysconfig/iptables had lines for https and for www with a --source option, which we comment out, and copy without it (our server is in an internal network behind a firewall, so we can allow unrestricted access). With this, we can monitor our server.
(:source lang=shell:)
# Allow these ports
-A INPUT -m state --state NEW -p tcp --dport ssh -j ACCEPT
# http and https is allowed for all nodes on the public subnet
#-A INPUT -m state --state NEW -p tcp --dport https --source 161.72.208.0/255.255.255.0 -j ACCEPT
#-A INPUT -m state --state NEW -p tcp --dport www --source 161.72.208.0/255.255.255.0 -j ACCEPT
-A INPUT -m state --state NEW -p tcp --dport https -j ACCEPT
-A INPUT -m state --state NEW -p tcp --dport www -j ACCEPT
  • Next step is create regular users, which is very easy. We just create a normal user in the frontend with the command useradd and then synchronize to all nodes with the command rocks sync users (the first time I run it, it complained about clock skews, which was solved by waiting some time, in the order of one or two hours. The second time I installed the cluster I didn't have this problem).
  • We are basically done with the basic cluster installation. We just need to verify that we can run (as a regular user) MPI programs. In the regular user account created above, we create a file named machines and try a basic MPI program.
(:source lang=shell:)
[angelv@xerenade ~]$ cat machines
xerenade
pvfs2-compute-0-0
pvfs2-compute-0-1
pvfs2-compute-0-2
[angelv@xerenade ~]$ /opt/openmpi/bin/mpirun -np 4 -machinefile machines /opt/mpi-tests/bin/mpi-ring
Process 0 on xerenade.ll.iac.es
Process 1 on pvfs2-compute-0-0.local
Process 2 on pvfs2-compute-0-1.local
Process 3 on pvfs2-compute-0-2.local
Process 1 on pvfs2-compute-0-0.local:successfully sent (1048576) bytes to id (2)
Process 3 on pvfs2-compute-0-2.local:successfully sent (1048576) bytes to id (0)
Process 3 on pvfs2-compute-0-2.local:successfully received (1048576) bytes from id (2)
Process 2 on pvfs2-compute-0-1.local:successfully sent (1048576) bytes to id (3)
Process 2 on pvfs2-compute-0-1.local:successfully received (1048576) bytes from id (1)
Process 0 on xerenade.ll.iac.es:successfully sent (1048576) bytes to id (1)
Process 0 on xerenade.ll.iac.es:successfully received (1048576) bytes from id (3)
Process 1 on pvfs2-compute-0-0.local:successfully received (1048576) bytes from id (0)
[angelv@xerenade ~]$
  • So all is good, and we can proceed with the ParaView installation.

Installation of ParaView

  • The stable version of ParaView at the time of writing was 3.6.1, which we download from here. We follow these instructions, which imply that we need to get Cmake and Qt. For Cmake we get version 2.6.4 and for Qt we get version 4.3.5.
  • Installation of cmake is just ./bootstrap ; make ; make install
  • Installation of Qt is simply ./configure ; make ; make install
  • For the installation of ParaView (assuming you unpacked the paraview-3.6.1.tar.gz in the /root directory, which creates the ParaView3 directory) we just do:
(:source lang=shell:)
[root@xerenade apps]# mkdir /share/apps/Paraview3-build                                                                                                      
[root@xerenade apps]# cd /share/apps/Paraview3-build/
[root@xerenade Paraview3-build]# export PATH="/usr/local/Trolltech/Qt-4.3.5/bin:$PATH"
[root@xerenade Paraview3-build]# ccmake /root/ParaView3/
  • When running ccmake, I modify the following options, so that they have these values:
(:source lang=shell:)
PARAVIEW_DATA_ROOT              /share/apps/Paraview-Data   -- (not sure if this is needed)
PARAVIEW_ENABLE_PYTHON          ON
PARAVIEW_USE_MPI                ON
CMAKE_INSTALL_PREFIX            /share/apps/Paraview3
  • Then after selecting the option generate and exit, we can just compile ParaView:
(:source lang=shell:)
[root@xerenade Paraview3-build]# make -j 4
[root@xerenade Paraview3-build]# export LD_LIBRARY_PATH="/usr/local/Trolltech/Qt-4.3.5/lib:$LD_LIBRARY_PATH"
[root@xerenade Paraview3-build]# make HTMLDocumentation
[root@xerenade Paraview3-build]# make install

Running ParaView

Depending on how you want to run ParaView, different settings could be possible. Our goal was to have the cluster only accessible to users via ssh. Some of the steps below could be easier if we left an X session open by one user, but we didn't like this idea very much, so the following will work assuming that no user is logged-in in any of the nodes in the cluster (which is convenient, since this is just the state you would get after rebooting. If you have a root session open in the frontend, you could log out and work remotely from now on).

  • The ParaView wiki tells you that configuring properly X connections is one of the most common problems with setting up a ParaView server. The way we solved it was with four basic scripts in /share/apps:
(:source lang=shell:)
[root@xerenade apps]# cat grant.X.access
#!/bin/bash
su - -c "xauth add :0.0 . `xauth -f /var/gdm/:0.Xauth list | awk '{ print $3}'`" $1

[root@xerenade apps]# cat rocks-grant
#!/bin/bash

rocks run host localhost "/share/apps/grant.X.access $1"
rocks run host pvfs2-compute-0-0 "/share/apps/grant.X.access $1"
rocks run host pvfs2-compute-0-1 "/share/apps/grant.X.access $1"
rocks run host pvfs2-compute-0-2 "/share/apps/grant.X.access $1"

[root@xerenade apps]# cat remove.X.access
#!/bin/bash
su - -c "xauth remove :0.0" $1

[root@xerenade apps]# cat rocks-remove
#!/bin/bash

rocks run host localhost "/share/apps/remove.X.access $1"
rocks run host pvfs2-compute-0-0 "/share/apps/remove.X.access $1"
rocks run host pvfs2-compute-0-1 "/share/apps/remove.X.access $1"
rocks run host pvfs2-compute-0-2 "/share/apps/remove.X.access $1"
  • grant.X.access will copy the appropriate magic cookies to the given user, so that he/she will be able to open an X application in the local monitor of one of the nodes. remove.X.access will just remove the magic cookie for that monitor. rocks-grant will get all the magic cookies for all the nodes and rocks-remove will delete them. Ideally you would like to write a script that will run in all the nodes of your cluster (without hardcoding their names as in the examples above, but this was just convenient for our small cluster). In any case, you should run the scripts grant.X.access or remove.X.access sequentially in each node and not in parallel, to avoid problems when locking the .Xauthority file.
  • Now it is very easy to add or remove permissions (to open X applications in the local monitors) for a given user (and we could easily integrate the granting and removing of permissions with a queuing system):
(:source lang=shell:)
[root@xerenade apps]# /share/apps/rocks-grant angelv

[root@xerenade apps]# su - -c "xauth list" angelv
xerenade.ll.iac.es/unix:0  MIT-MAGIC-COOKIE-1  9aa26f848641255407e8f8b885c7dbe9
pvfs2-compute-0-0.local/unix:0  MIT-MAGIC-COOKIE-1 307ba11af5ee1bec5598e9bca8b11e7c
pvfs2-compute-0-1.local/unix:0  MIT-MAGIC-COOKIE-1 d14de616f106dc3f17cda434b7d85e9e
pvfs2-compute-0-2.local/unix:0  MIT-MAGIC-COOKIE-1 bb7c277a195fd04caa35ce273a3f2bc1

[root@xerenade apps]# /share/apps/rocks-remove angelv

[root@xerenade apps]# su - -c "xauth list" angelv

[root@xerenade apps]#
  • Now we can verify that after running rocks-grant, the given user can open an X application in the local monitor. For example:

Before running rocks-grant

(:source lang=shell:)
[angelv@xerenade ~]$ emacs -display :0
Xlib: connection to ":0.0" refused by server
Xlib: No protocol specified

emacs: Cannot connect to X server :0.
Check the DISPLAY environment variable or use `-d'.
Also use the `xhost' program to verify that it is set to permit
connections from your machine.

After running rocks-grant (connected via ssh from another machine, and opens a window in the local monitor of the cluster)

(:source lang=shell:)
[angelv@xerenade ~]$ emacs -display :0
  • So now is time to run glxgears (as suggested in the instructions) in parallel. In our cluster we found the problem that when running glxgears, it complained that the "GLX" extensions were missing:
(:source lang=shell:)
[angelv@xerenade ~]$ glxgears -display :0
Xlib:  extension "GLX" missing on display ":0.0".
Error: couldn't get an RGB, Double-buffered visual
[angelv@xerenade ~]$
  • This was easily solved adding the line Load "glx" in the file /etc/X11/xorg.conf, and restarting the X server in all the nodes with the command rocks run host gdm-restart
(:source lang=shell:)
[angelv@xerenade ~]$ grep -C 3 glx /etc/X11/xorg.conf

Section "Module"
        Load  "vnc"
        Load  "glx"
EndSection

Section "InputDevice"
[angelv@xerenade ~]$
  • Now (after running rocks-grant again, since each time that the X server is started, the magic cookies change), glxgears is no problem (though VERY slow, this is a very old test machine):
(:source lang=shell:)
[angelv@xerenade ~]$ mpirun -np 4 -machinefile machines glxgears -display :0
631 frames in 5.3 seconds = 118.530 FPS
630 frames in 5.3 seconds = 118.222 FPS
632 frames in 5.3 seconds = 118.487 FPS
390 frames in 6.0 seconds = 65.440 FPS
  • You could verify that glxgears is really running in all nodes:
(:source lang=shell:)
[angelv@xerenade ~]$ rocks run host 'ps -fea | grep glxgears' | grep "display"
pvfs2-compute-0-0: angelv   20725 20724  0 13:44 ?        00:00:00 glxgears -display :0
xerenade: angelv    4192  3762  0 13:44 pts/5    00:00:00 mpirun -np 4 -machinefile machines glxgears -display :0
xerenade: angelv    4198  4197  0 13:44 ?        00:00:00 glxgears -display :0
pvfs2-compute-0-2: angelv   20033 20032  0 13:44 ?        00:00:00 glxgears -display :0
pvfs2-compute-0-1: angelv   20136 20135  0 13:44 ?        00:00:00 glxgears -display :0
[angelv@xerenade ~]$
  • We are almost there, and now we just need to do the final test, in which we will have a ParaView client connected to the ParaView server in the cluster. In the server we run ParaView in parallel (make sure that the name of the frontend is the first one in the machines file):
(:source lang=shell:)
[angelv@xerenade ~]$ mpirun -np 4 -machinefile machines /share/apps/Paraview3/bin/pvserver -display :0
Listen on port: 11111
Waiting for client...
  • In another workstation I download the "Linux x86, 64 bit" ParaView software, and running it gives no trouble:
(:source lang=shell:)
angelv@vaso:~$ paraview-3.6.1-Linux-x86_64/bin/paraview
  • We now have to define the server to connect (File -> Connect), and we specify a new connection, with Host being the name of your server, and Startup Type set (for the moment) to Manual. We click to connect, and then in the ParaView server we see the last line appearing, showing a partial success.
(:source lang=shell:)
[angelv@xerenade ~]$ mpirun -np 4 -machinefile machines /share/apps/Paraview3/bin/pvserver -display :0
Listen on port: 11111
Waiting for client...
Client connected.
  • We try rendering a sphere in the ParaView client GUI (Sources -> Sphere, and we select Theta Resolution and Phi Resolution to 1024), but after clicking Apply we only see a quarter of the sphere and some error messages appear in the server terminal:
(:source lang=shell:)
[angelv@xerenade ~]$ mpirun -np 4 -machinefile machines /share/apps/Paraview3/bin/pvserver -display :0
Listen on port: 11111
Waiting for client...
Client connected.
ICET,0:ERROR: Sizes of src and dest do not agree.
ICET,0:ERROR: Sizes of src and dest do not agree.
  • This (which probably doesn't happen in most settings) was due to different screen resolutions:
(:source lang=shell:)
[angelv@xerenade ~]$ rocks run host "xdpyinfo -display :0 | grep -C 1 '#0'"
xerenade:
xerenade: screen #0:
xerenade:   dimensions:    1280x1024 pixels (342x271 millimeters)
pvfs2-compute-0-0:
pvfs2-compute-0-0: screen #0:
pvfs2-compute-0-0:   dimensions:    800x600 pixels (271x203 millimeters)
pvfs2-compute-0-2:
pvfs2-compute-0-2: screen #0:
pvfs2-compute-0-2:   dimensions:    800x600 pixels (271x203 millimeters)
pvfs2-compute-0-1:
pvfs2-compute-0-1: screen #0:
pvfs2-compute-0-1:   dimensions:    800x600 pixels (271x203 millimeters)
[angelv@xerenade ~]$
  • To get the frontend in line with the other nodes, I add the Modes "800x600" line to /etc/X11/xorg.conf, and then restart the X server.
(:source lang=shell:)
[root@xerenade ~]# grep -C 3 -i modes /etc/X11/xorg.conf
        SubSection "Display"
                Viewport   0 0
                Depth     24
                Modes "800x600"
        EndSubSection
EndSection

[root@xerenade ~]#
  • With this in place, we can rocks-grant permission to our regular user again, restart the server and the client, and now when rendering the sphere as for the steps above, there are no errors, and everything works as expected!!

Comments or suggestions to improve this guide can be sent to

Section: HOWTOs

edit · print · PDF
Page last modified on November 17, 2009, at 10:14 AM