Here is a bit of interesting job scheduling magic when you run two web app containers with a single data container.
Have a look a this command line action:
Last login: Thu May 21 03:06:57 2020 from 159.192.33.10
srw-rw---- 1 root docker 0 May 5 07:38 /var/run/docker.sock
lrwxrwxrwx 1 root root 44 May 15 09:16 /var/run/nginx.http.sock -> /var/discourse/shared/socket/nginx.http.sock
/var/discourse# docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
0d7c881b4838 local_discourse/socket "/sbin/boot" 6 days ago Up 6 days socket
b297dd5caa6e local_discourse/socket2 "/sbin/boot" 8 days ago Up 8 days socket2
640e5c80be5f local_discourse/data "/sbin/boot" 2 weeks ago Up 2 weeks data
/var/discourse# find ./shared/socket2/uploads | wc -l
198372
/var/discourse# find ./shared/socket/uploads | wc -l
198462
/var/discourse# cp -rf ./shared/socket/uploads/* ./shared/socket2/uploads
/var/discourse# find ./shared/socket2/uploads | wc -l
198555
/var/discourse# find ./shared/socket/uploads | wc -l
198462
/var/discourse# cp -rf ./shared/socket2/uploads/* ./shared/socket/uploads
/var/discourse# find ./shared/socket2/uploads | wc -l
198555
/var/discourse# find ./shared/socket/uploads | wc -l
198555
What does this mean and why should any Discourse sys admin be interested?
In this configuration, there are two web application containers running at the same time, and both of them use Redis / Sidekiq to pull down images and store them locally and both optimize images.
So, over time, the files in the upload directories of each app instance will diverge (as expected) as Redis / Sidekiq does their background, job scheduling magic.
Also, naturally, the active container (the one current active and live for users) will have new avatars and new uploads; and at the same time the background job scheduler is also optimizing images.
At the same time, the other container, call it the "background container" for now, is also optimizing.
This is why the uploaded files with diverge over time. This is a "good thing" and have some key advantages I will discuss later.
Because we consider both web app containers as "hot standby, equal opportunity" containers, we can, at any time, switch between the two containers with a simple ln -sf command linking the unix domain sockets in the containers to the user-facing ngnix reverse proxy.
So, by keeping the uploads in sync by copying the files back and forth between the upload directories (which I just did manually to illustrate the point), we keep these two web app containers in sync (the uploads).
We also do this when we switch containers, for example let's say we were running on the socket2 container and decided to rebuild socket and then run on socket, moving socket2 to the background. At that time, the uploads will be out of sync, so we sync them.
Both of these containers will continue to schedule jobs, optimizing images in the backgound, even though one container is "actively managing user requests" and the other container is "hanging out in the background".
Note. We could move the uploads to a shared volume and use a symbolic link to link the uploads from one of the container's shared folder to the other container's shared folder; but in this example, that link has not been created because I have not tested shared the uploads directory between two web applications in production yet.
The magic of Redis /Sidekiq job scheduler is very interesting, when you think about it.
I will write a bit more on this later with real-world examples as they appear.
Anyone else use Redis / Sidekiq and have any interesting, real-world examples of parallel background processing / job scheduling to share? Please do!