Is there some was to test if opening with the O_DIRECT flag will work (without writing a C program)?
Unfortunately no. If you do manage to open anything with O_DIRECT though it will be very inefficient, may I ask the purpose here? There may be better ways to do what you want(posix_fadvise, etc.)
When moving data in XFS file systems (using custom software), I can drive the disk arrays harder with non-buffered I/O, thus I like to open with O_DIRECT. That is, direct I/O is *way* faster in this case.
One of the checks I would like to do in a wrapper script is check if direct I/O support is available in the underlying file system to determine what flags to pass to the custom software.