How to Install and Run Faucets Overview There are possibly several computers involved since each component listed below can run on a separate machine. * MySQL for CS * CS * MySQL for CM * CM * CD * CC, guiC * Appspector Installing Faucets requires several steps. * Install MySQL for CS * Install CS * Install MySQL for CM * Install CM * Install CD * Install CC, guiC Running faucets * start CS MySQL, CS * start scheduler MySQL, scheduler. test with fjobs, etc. * start CD (registers itself w/CS) * run the client ================================================================ * Install MySQL for CS Download and install MySQL from ?? Use the .tar.gz version so that you can more easily run it as a non-root user. Follow the instructions in the INSTALL.TXT document very carefully; it is not very user-friendly. If you have questions, browse the documentation at mysql.com. Start mysql % safe_mysqld --log --port=20001 & Start a mysql client: % mysql --no-defaults -u root -p -P 20001 and type in the following commands to create the needed database and tables. mysql> create database faucet; mysql> use faucet; mysql> create table Cluster ( domainName text not null, port int not null, status text ); mysql> create table users ( userid text not null, password text not null ); To grant the CS permission to access the database as user "cs" with password "password", do the following: (There's probably a better way to do this; if you find out, let me know.) mysql> grant all on *.* to cs identified by "password"; mysql> grant all on *.* to cs@hostname_of_mysql_server identified by "password"; mysql> grant all on *.* to cs@localhost identified by "password"; Now you will need to create some faucets usernames and passwords. These represent faucets users who will submit jobs to the faucets system. mysql> insert into users (userid, password) values (faucet_user, "password"); mysql> quit; To shutdown the mysql daemon: % mysqladmin --no-defaults -u root -p shutdown ================================================================ * Install CS Unpack the faucets code. This will create a directory called "faucets". % cd faucets % make Edit faucets/cs/db.properties and update the hostname of the server on which mysql is running, the mysql username (e.g. "cs"), and the password (e.g. "password"). (Note: The username and password should match the commands you gave above to the MySQL server.) You will need a JDBC driver for mysql. Download one from http://mmmysql.sourceforge.net/. We have tested it with version 2.0.8. To start the CS: % cd faucets % java -cp .:/path/to/mm.mysql-2.0.8-bin.jar TheServer To stop the CS: Just press Ctrl-C where the CS is running. ================================================================ * Install MySQL for CM The installation of MySQL is the same as described above. But the commands to create the databases and tables differ. ================================================================ * Install CM Requires charm++, scheduler, fjobs, etc. ================================================================ * Install CD The CD assumes the CM is running on the same computer. It will use frun, fsub, etc. commands to issue jobs to the CM. Read the CM installation file to install the CM and after the CM has been installed, set the environment variable SCHEDULER_PATH to //pgms/scheduler, AND add $SCHEDULER_PATH/qsystem to your path. Unpack the faucets code. This will create a directory called "faucets". % cd faucets % make To start CD: % cd faucets % java -cp .:./common/TB.jar cd.ClusterDaemon /tmp/ > #To run CD > cd faucets/cd > setenv CLASSPATH ${CLASSPATH}:/home/net/skumar2/biocore/collab/ #> setenv PATH ${PATH}:/home/net/skumar2/qsystem/bin // not necessary any more > java cd.ClusterDaemon -p ================================================================ * Install CC Unpack cc code % cd cc % make To run Faucet commandline client (make sure that the CLASSPATH points to faucets directory) % java cc.FaucetCLI will print out all the available options, as given below java cc.FaucetCLI [-input file1,file2,...,filen] [] This option submits a job with the input files provided by the -input option. File names are seperated by ',' and should not have spaces, file names should be relative or absolute paths of the files Returns the job id of the job java cc.FaucetCLI ListJobs Lists all job of the current user. This option is currently not supported ! java cc.FaucetCLI ListApps Lists the applications that can be run by all the Cluster Daemons, only applications that are registered at the daemon can be run by the user. java cc.FaucetCLI GetStatus Gets the status of the job give by job-id. Possible values returned are: QUEUED : job is queued for execution RUNNING : job is running FINISHED: job finished execution UNKNOWN : java cc.FaucetCLI ListFiles Lists the files generated by the application, out and err are the standard output and standard files generated by the system. java FaucetCLI GetFile Gets the file from the CD into the present working directory. java FaucetCLI GetFile Gets the file from the CD into the directory given by path. Path can be absolute or relative. java FaucetCLI KillJob Kills the job submission. Sample runs: (Assuming that the central server is running on cool2.cs.uiuc.edu at port 4000) java cc.FaucetCLI cool2.cs.uiuc.edu 4000 listapps userid :root // enter username as present in the users table, created during database setup PASSWORD for root:test // enter password CM cool2.cs.uiuc.edu 4999 // registered clusterdaemons are listed CM cool2.cs.uiuc.edu 5000 end Primes Demo //applications that can be run by all Cluster Daemons ls NAMD 2.1 Jacobi-Gauss Plate Demo java cc.FaucetCLI cool2.cs.uiuc.edu 4000 "NAMD 2.1" -input apoa1.namd,apoa1.pdb,apoa1.psf,par_all22_popc.xplor,par_all22_prot_lipid.xplor -maxpe 24 -minpe 1 apoa1.namd userid : root PASSWORD for root: test CM cool2.cs.uiuc.edu 4999 CM cool2.cs.uiuc.edu 5000 end make bids:{CM cool2.cs.uiuc.edu 5000=yes, CM cool2.cs.uiuc.edu 4999=yes} Preparing to transfer file apoa1.namd... Store successful., Got ACK Preparing to transfer file apoa1.pdb... Store successful., Got ACK Preparing to transfer file apoa1.psf... Store successful., Got ACK Preparing to transfer file par_all22_popc.xplor... Store successful., Got ACK Preparing to transfer file par_all22_prot_lipid.xplor... Store successful., Got ACK Job id is cool2.cs.uiuc.edu:5000/3382254831828920918 java cc.FaucetCLI getstatus cool2.cs.uiuc.edu:5000/3382254831828920918 userid : root PASSWORD for root: test RUNNING java cc.FaucetCLI getmonitorurl cool2.cs.uiuc.edu:5000/3382254831828920918 userid : root PASSWORD for root: test http://finesse.cs.uiuc.edu/~skumar2/cgi-bin/AppMonitor.pl?ID=1017807530847 netscape http://finesse.cs.uiuc.edu/~skumar2/cgi-bin/AppMonitor.pl?ID=1017807530847 # to monitor the job. java cc.FaucetCLI killjob cool2.cs.uiuc.edu:5000/3382254831828920918 userid : root PASSWORD for root: test job killed 3382254831828920918 java cc.FaucetCLI listfiles cool2.cs.uiuc.edu:5000/3382254831828920918 userid :root PASSWORD for root:test apoa1.namd apoa1.pdb apoa1.psf par_all22_popc.xplor par_all22_prot_lipid.xplor out1017807590870 err1017807590870 java cc.FaucetCLI getfile cool2.cs.uiuc.edu:5000/3382254831828920918 err1017807590870 userid :root PASSWORD for root:test Preparing to receive file... File received. java cc.FaucetCLI getfile cool2.cs.uiuc.edu:5000/3382254831828920918 err1017807590870 tmp userid :root PASSWORD for root:test Preparing to receive file... //file downloaded in tmp directory of the present working directory File received. -f option: ---------- -f option allows user to specify all the required parameters in a file. command line client reads data from the file, runs the job and loads the "out*" and "err*" files in the specified directory after the job is finished. name of the parameter file should be of the form .properties it should contain data in the form = where can be from the following list: appname - name of the application, ex: NAMD 2.1 (quotes (") not required) hostname - central server name port - central server port minpe - minimum number of processors maxpe - maximum number of processors interval - time interval to check whether job is finished or not inputFiles - input files separated by commas (should contain either absolute path or the path accessible from the working directory) directory - directory where output files must be downloaded (default: all files are downoaded) parameters - any additional parameters, ex: apoa1.namd in case of NAMD 2.1 Sample run: java cc.FaucetCLI -f namd //namd.properties file should be present in the present working directory namd.properties found. CM cool2.cs.uiuc.edu 4999 CM cool2.cs.uiuc.edu 5000 end make bids:{CM cool2.cs.uiuc.edu 5000=yes, CM cool2.cs.uiuc.edu 4999=yes} Preparing to transfer file /home/net/bandhaka/faucets/apoa1/apoa1.namd... Store successful., Got ACK Preparing to transfer file /home/net/bandhaka/faucets/apoa1/apoa1.psf... Store successful., Got ACK Preparing to transfer file /home/net/bandhaka/faucets/apoa1/apoa1.pdb... Store successful., Got ACK Preparing to transfer file /home/net/bandhaka/faucets/apoa1/par_all22_popc.xplor... Store successful., Got ACK Preparing to transfer file /home/net/bandhaka/faucets/apoa1/par_all22_prot_lipid.xplor... Store successful., Got ACK Job id is cool2.cs.uiuc.edu:5000/4540828302748155698 http://finesse.cs.uiuc.edu/~skumar2/cgi-bin/AppMonitor.pl?ID=1017816431633 // after some delay i.e. after the job is finished Job: 4540828302748155698 finished apoa1.namd apoa1.psf apoa1.pdb par_all22_popc.xplor par_all22_prot_lipid.xplor out1017818849455 err1017818849455 asking for file out1017818849455 Preparing to receive file... File received. asking for file err1017818849455 Preparing to receive file... File received. files saved =============================================================== * Install and run gui Unpack gui code % cd gui % make to run the gui window: Authentication: -------------- % make run This pops up the login window Enter Login Name and Password (as registered with CS, in the mysql database above) Click on "submit" button OR press "enter" key if an error dialog box pops up saying "Invalid Password. Try Again.", then authentication was not successful. check your login name and password and try again. if authentication is successful, a new window to "submit and monitor jobs through faucets" appears. Now the user can submit jobs by entering the required data. List runnable applications: -------------------------- "List Application" button lists all the applications currently runnable by all the available Cluster Daemons. These appear in the "Applications" ListBox. Select the application to submit: -------------------------------- Select the application u want to run by clicking on the corresponding name in the list. Specify input files: ------------------- For whatever application selected, choose the files that are required to run it, using the "Choose Files" button. Clicking on "Choose Files" button pops up a FileOpen DialogBox, which can be navigated to choose the required files. To select multiple files, hold down "ctrl" key on the keyboard while clicking on the filenames. Click "OK" after selection is complete. Click "Cancel" to cancel the selection. Specify additional parameters: ----------------------------- maxpe : enter the maximum number of processors to be used to run the job minpe : enter the minimum number of processors to be used to run the job command line options: additional job specific options; ex: configuration files etc. Submit Job: ---------- After specifying the parameters properly, click on the "Submit" button, to submit the selected job. The status of the submission can be seen in the status panel. A successful submission would result in statements like the following: CM cool2.cs.uiuc.edu 4999 CM cool2.cs.uiuc.edu 6999 end make bids:{CM cool2.cs.uiuc.edu 6999 Job id is cool2.cs.uiuc.edu:4999/5912403195047484260 Monitor Job: ----------- After the job is submitted, it can be monitored by clicking the "Monitor" tab. The table should list the name of the submitted job and the url where it can be monitored. Click on "Monitor Job" button; it pops up a window which displays the status of the job, its output and graphs to monitor the running of the job. Kill Job: -------- A running process can be killed by clicking on the "Kill Job" button. status is shown in the status panel. Successful killing of a job would result in a status like this: job killed 5912403195047484260 Killed Job: NAMD 2.1 Download files from the server to the local directory: ----------------------------------------------------- Click on "List Files" to pop up a window "List of Files" which displays the list of files at the server side related to the present job. click on the names to select the files. Multiple selections are possible by holding down the "ctrl" key. After selecting the files, click the "Download" button. This opens a "Save Dialog Box" which can be navigated to select the directory to store the downloaded files. choose the directory and click ok to complete downloading the files. Successful download results in a dialogbox which says "Files downloaded successfully". Note: ---- The CS server contacted by the gui is listed in the cs.properties file present in the directory faucets/gui. AppMonitor still uses hardcoded host and port : "finesse.cs.uiuc.edu" & 6234 ================================================================ * Install Appspector Appspector is used to monitor the status of the submitted job. Monitoring of the job can be done in three ways: 1. through a webbrowser by connecting to the url returned when get-appspector-url command is executed. 2. through the gui, by clicking the monitor button in the display panel. 3. by running AppMonitor class on the command line, passing as parameters , and the appspector id of the job. the appspector id is the number at the end of the url returned from the server. The AppMonitor class is responsible for the popping up of a window to monitor the job, given the appspector server name, port number and appspector id of the job. Usage : java AppMonitor