The Contact form in Joomla! uses the mail server settings to send the contact request email to the site administrator. However, when using SMTP servers (I've tried both Yahoo! and GMail SMTP servers), it does not work, as the email address of the sender is not recognized by the SMTP server; in fact, the only way to make the SMTP server work in the Contact form, is using your own email address to request a contact (an hardly useful feature).
You can avoid all these troubles with a Joomla! component named DFContact (that is the component used in this site's Contact page, so you can check quickly how it works). DFContact packages the information from the Contact form into an email sent from your own email address that your SMTP server will happily accept and deliver.
This is a sample email: as you can see, the sender and the addressee emails match, the email address of the person requiring the contact is inside the email body so you cannot just click on Reply to answer.
You can download DFContact from here.