The pensions and Isas of millions of British savers have been dented this week by the revelation that a middle-aged man thousands of miles away had an affair.