Add a local cluster option to execute on standard linux or mac machine #164

Activity

1. ieee8023 commented on May 4, 2017

   

   ieee8023

   on May 4, 2017

   Author

   More actions

   More options:

   xargs or make http://coldattic.info/shvedsky/pro/blogs/a-foo-walks-into-a-bar/posts/7

   at and batch: https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/3/html/System_Administration_Guide/s1-autotasks-at-batch.html

   I think make is a good solution. It has a thread pool built in. Here is an example job makefile.

   default: job1.log job2.log job3.log
   job1.log:
   	sh job.sh 1 > job1.log
   job2.log:
   	sh job.sh 2 > job2.log
   job3.log:
   	sh job.sh 3 > job3.log
   

   • You can launch make it make -j 2 to have thread pool of size 2.
   • It depends on the log files, so if a job has run it will not run again if you run the makefile again.
   • The makefile can be generated as the submission script for a cluster.
   • Make can be run using at so it is a background process on the machine and out of interactive mode.
2. bouthilx commented on Jul 11, 2017

   

   bouthilx

   on Jul 11, 2017

   Collaborator

   More actions

   @mgermain Does the Trello link mean you did some work on your side? We are considering this issue for a CCW ticket.
3. mgermain commented on Jul 11, 2017

   

   mgermain

   on Jul 11, 2017

   Member

   More actions

   Just removed the trello link. It was not related. No work has been done for local a local queue and I think there are more pressing issues even if this one would be relly nice to have.
4. bouthilx commented on Jul 12, 2017

   

   bouthilx

   on Jul 12, 2017

   Collaborator

   More actions

   Which issue would you consider more important? A student was asigned the slurm backend ticket. I though he could add support for graham and cedar afterwards, but meanwhile I would have given this ticket or one you find more important to another student.
5. mgermain commented on Jul 17, 2017

   

   mgermain

   on Jul 17, 2017

   Last edited by bouthilx

   Member

   More actions

   There are a lot of almost finish, small or low hanging fruit that we could finish before adding another backend which is a much more involved task.

   ToFinish/Review/Merge PR

   #156 - Add job dependency to autoresume feature
   #157 - Don't use -V
   #113 - Write Documentation
   #42 - Add support for memory

   Quick&Small

   #140 - Add Coloss support
   #96 - When msub failed to launch a job we should have some feedback

   Important and overdue

   #108 - Support custom queues
   #62 - Add a ~/.smartdispatch config folder
   #7 - Add more types of folded arguments