You may be affected by this bug.
If this is the case, you have the wrong ssh-agent set as active. You can check this with the command
$ echo $SSH_AUTH_SOCK
If the agent value begins /tmp/ssh, you have the commandline ssh-agent acting as your agent.
As a workaround, you can use the old style shell command
$ ssh-add
to add you key to the keyring. This does not work well with all graphical programs, however.