Hi,
I have searched the forums and could not find a relavant thread discussing my use case, hence the new post.
Basically am trying to pass on work to dummy worker instances from controller which will pass on work to workers (client)
To make use of host capacity, am planning to serialize actual command structure ( the structure that contains the details of work, how to execute, which log file etc ) and dump it to NFS directory that is shared by hosts in which both worker and controller runs.
In a way, it could be explained as -
1) controller creates the structure serializes and dump it to one of the NFS directory as a file
2) worker waits for the file, reads the structure and deserialize it and start executing the work as specified in the structure
I actually don't like this approach as its not crash resistant, not scalable and have been thinking to use message queues (POSIX) to implement this asynchronous communication between worker and controller. But I don't know how to create a message queue that can be accessed across process in multiple hosts.
If that is possible, then its very good, but how?
If not, is there a way to dump message queue as a serialized structure to NFS directory and the worker can deserialize the structure and read that again.
I know, this also sounds messy as the very first point is simultanity in read and write to MQ is lost with this approach.
Also, if you could suggest alternative methods that would be awesome.
Other methods, I could think of is
- socket conn establishment and communicating via that - much of handling and house keeping
- use Amazon SQS, high available centralized queue - I need to pay and its a big setup for my small use case run in 4 hosts shared via 1 NFS mount.
Many thanks for suggestions in advance. 