The environment is Java/Windows. The program keeps near real-time state in memory cache, which is updated by multiple sources, size of the cache is roughly 500 MB, frequency of updates is ~ 20 per second. I am looking into different ways to keep current snapshot of the memory on the disk for a) planned program restarts and b) possibility of failover. Requirement for downtime is not more than 2 minutes.
So far two options are discussed: 1) write serialized objects into plain files and 2) use RDBMS (MS SQL Server)
Both have pros and cons: for (1) there is considerable effort to re-populate cache from disk, but it should be the fastest; for (2) the programmers plan to use ORM, such as MyBatis and they say it will cut programming effort, although we are adding vulnerabilities of frequently updated RDBMS and speed needs to be proved to be fast enough.
What would gurus do?
Please don't suggest Hybernate, TopLink etc... to use as cache repository.
If it wasn't Java, I'd tell you to memory-map the file. It's a fundamental feature of any modern OS which Java lacks the ability to directly use. Without that, you're stuck with the options you mentioned.
How long does it take to recreate your data when it's not cached?
If that's anywhere near as fast as recreating the cache, the added overhead of recreating the cache is wasted.
Also, how much performance are you going to lose when you have to keep cache updates synchronized with a persistent data store? If you need reliability, modifications to the cache can't return until data is sync'd to disk.
Nevermind the cost of writing and maintaining the cache recreation code. What happens when that gets buggy, such as not remaining consistent, or improperly syncing changes to disk?
And FWIW, writing complex code makes it a whole lot harder to meet reliability requirements.
I have exactly same question - can we re-start app and read in this FileChannel-based storage into the memory and bring to life the objects in my cache ?
POSIX message queues persist over crash/reboot, until mq_unlink is called. They can be re-opened in the state they were. They are implemented as files without file semantics.
The only thing that CAN persist over a crash/reboot is a physical object - like bytes on a disk, flash disk, tape, etc.
SYSV and POSIX IPC objects are kernel persistent. They survive a program crash.
Not a system crash or reboot.
It will provide interface to file based b-tree or hash table where you can cache raw objects without serialization. I use it often in C/C++ for file based caching since its pretty damn fast and has superb scaling even on caches with sizes of many terabytes.
A region of a file may be mapped directly into memory; for large files this is often much more efficient than invoking the usual read or write methods.