scp without password for destination server

I am using the below command to scp the file to destination server ( $server ).
The ssh keys have been created between the source and destination servers. Still I am getting password authentication after executing the below scp command:

scp $Local_dir/$file_pattern* $username@$server:/$ftp_dir/

Pls suggest

What happens if you try a simple ssh connection?

If you get prompted for a password prompt for ssh also, use several "-v" options to get verbose output:

ssh -v -v -v ...

That should tell you what's wrong.