James has now broken his silence on the allegations saying he considered it 'unfair' for just one of them to discuss their marriage, yet insisted he did not have an affair.