As the ledger which is recording all the bitcoin transactions is made truly public, the privacy is at stake. Anybody in the world would be able to know who paid whom? The traditional banking system is able to maintain this kind of privacy by keeping its records confidential.
Privacy in Bitcoin system is achieved by a different strategy. Note that we said that the sender of a bitcoin needs to know whom to pay. So he asks for the public key of the vendor to which he desires to make the payment. This public key can be anonymous.
In the sense, as a vendor of some services, when somebody asks you where to send the payment, you would simply send him your public key. The association of this public key with you is not recorded anywhere in the ledger. That way anybody outside of this transaction would only know how much money is transacted and to which public key the money is paid out.
To achieve a higher degree of privacy, for every transaction, you may generate a new private/public key for each transaction so that multiple transactions made by you cannot be grouped together by a third party. For an outsider, this would simply mean that multiple transactions of smaller values were made and they will never will be linked to a common source.
Lastly, any online internet based system is vulnerable to abuses. I will now describe few possible types of attacks on Bitcoin system and how those are mitigated.