Please confirm that you're getting this error message while trying to CREATE the backupserver.
The reason I ask is because you mention seeing these messages in the backupserver errorlog, but you wouldn't be getting any messages in the backupserver errorlog unless you were able to successfully install a backupserver.
Also, what is your dataserver/backupserver version strings? (dataserver -v , backupserver -v)
---------------
A quick google search turns up Note 1963584:Open Server Error 16400 in ASE Backup Server log when dumping a database. [References SAP note 1963584.]
- same errors as you're seeing, but errors generated while attempting to perform a dump command (ie, backupserver has been installed and is up and running)
- appears to be an issue when "net password encryption" is enabled/true for SYB_BACKUP; if you have net password encryption enabled ... what happens if you disable it?