Strange, your example works correctly for me.
In sed you can combine commands, so you can change a little, for example:
sed 's/^\"//; s/\"$//' file
There is no need for the g modifier because there is only one beginning and one end in a line.
Adding possible spaces to remove:
sed 's/^\s*\"\s*//; s/\s*\"\s*$//' file
Now let's try to do everything in one command using the algorithm proposed by @RavinderSingh13
sed -r 's/^\s*\"\s*|\s*\"\s*$//g' file
If we add the option -i then create a new file with the same name and delete the old one:
sed -ri 's/^\s*\"\s*|\s*\"\s*$//g' file
In this case, the -i option should be the last