No, this isn't what you want. The double quotes are there in the last line because the 3rd field in that line contains an unmatched single quote. Before you strip the double quotes you need to look at the source of your input data and fix whatever is causing the mismatched quotes to appear in your file.
OK. But, be VERY careful in handling the fields you get from splitting lines in this file. Unmatched quotes have a tendency to cause cause lots of problems in scripts written by people who don't meticulously use quotes appropriately.
Note that the sed scripts that pamu and Scrutinizer have supplied will remove any double quotes at the start of any line and at the end of any line even if they don't form a matched pair. If you only want to remove matched pairs (which is what you requested), you probably want something like:
sed 's/^"\(.*\)"$/\1/' file
instead. Even then, if you have an input line like:
"01";'This is 2nd field';"This is 3rd field'"
you will destroy matched quotes leaving you with unmatched double quotes in the 1st and last fields and well as the unmatched single quote in the last field.