Prior to implementation of Veeam as our primary solution for backing/replicating stuff, we used to have similar issues with vSphere Replication.
In our case, the problem was related to VR ip configuration. We happened to assign such an IP address to VR server that was already being used in our environment. So, If were you, I would doublecheck VR IP settings and ensure that the corresponding IP address has been already taken.
Thanks to Veeam, nowadays everything works as a charm.
Hope this helps.