Check out the mt command - doing a forcereserve may work
OR
if this is a single tape drive you might be able to eject the tape manually - not sure about a jukebox (turning off a jukebox or opening it up and ejecting manually may work)
OR
theory: try the eject command with -f - not sure if it will take a tape device and no way for me to test.
what OS and version and type of tape drive may get more answers. What software is being used to control the device?
If you can't mt the tape device I wonder if it would be possible to use camcontrol to turn off the SCSI tape drive and force the device to just error out?
Until the driver finishes with the current operation, that process is not going anywhere. Cycling power on the tape drive may abort the operation. Other than that your options are reboot or wait until the operation finishes.
camcontrol is a SCSI control program. After a few cursory Google searches it would appear that it is only available under *BSD. It gives the root user the ability to control a SCSI device and shutdown/powerup/add/detch/etc.
Maybe there is something equivilent under SCO for SCSI sub-system control?
But as Perderabo mentioned, if the TBU is external you could just cut power to it and let the process err out (which the hopes that this would not be a cascading failure to some other sub-system, but that is probably just my own paranoia kicking in).
If the TBU is external, then I agree with Perderabo. Shutting down the power may abort the operation and it might be possible to reset the TBU from the OS.