Seeking tool/solution for instant replication between 2 Linux systems

Hello,

We has some devices, which is loaded with customized Linux. That means, it will not have all libraries and packages. It generates some data and that has to be copied to one RHEL 7.9 VM. Fluentd is running on this, which is responsible for uploading data to AWS S3 bucket. I have setup rsync on RHEL server, which sync with those devices every 30 seconds and copy data, which is new. But there is latency and delay, as rsync can't be instant. I need to run the sync every some seconds.
Apps guys don't want to use NFS, as they have some limitations from app side.

Here is flow -->
devices (1, 2, 3, 4...) ---(latency)--> RHEL (with fluentd) ---(upload to) ---> AWS S3 bucket.

I am trying to find other ways or tools which are linux based and which I can setup between devices and RHEL server and can instantly replicate data. I need some advice here.

(These devices are supported by vendor. To keep the data replicated instantly, they have bigger solution and expensive too, which management don't want to adopt for now)

Thanks

btw, there's latency in all of those flow components, some more obvious than others :smiley:

checkout lsyncd it might meet your criteria

1 Like

My guess is that fluentd is used for some kind of parsing / transforming data ?

How about using RHEL as a proxy to AWS bucket ?
Something in the line of install minio client on 'devices' (one GO binary) and set bucket creds, setup HTTP(S) proxy as RHEL server and upload to AWS s3 bucket using that proxy.

Or remove RHEL from the equation completely...
What kind of data are those devices sending and can it be transformed on the source 'devices', removing RHEL/fluentd part all together ?

Regards.

1 Like

Thanks munkeHoller and Peasant.
Using RHEL server as proxy can be a good idea, but I need to check if those devices are capable of adjusting minio client. I will get more details and see, if this solution is adoptable. Thanks for tips.

2 Likes