What I did:
1. change max shared memory:
vi /etc/sysctl.conf
2. run /sbin/sysctl -w kernel.shmmax=10179869184
3. Change sybase Max Memory
then I reboot linux by reboot.
As I it failed, I put all back:
1. vi /etc/sysctl.conf
2. /sbin/sysctl -p
3. vi ASEServer.cfg in /sybase/ASE-12_5/ to change max memory back
then reboot linux and start sybase, get same error. Not sure why. How to put everything back? help please. thanks.