Replication Server failed with fatal error while allocating memory. Most likely, there is not enough memory. Replication Server is going to abort.
If you are working on SAP ASE(Sybase) replication server, there is likelihood of facing this error which states “Replication Server failed with fatal error while allocating memory. Most likely, there is not enough memory. Replication Server is going to abort.”
We faced this issue when we added a MSA(Multi-Site Availability) to the existing set-up of warm standby. All the configurations went fine and database replication started smoothly until late evening. Replication server suddenly crashed within few minutes when EOD(End Of Day) started with above error message. Our replication server was not updated in terms of hardware/software from quite sometime as it was working smoothly and as most of Sybase DBA does – don’t touch replication server if all is well.
Replication server was having very limited memory and number of cores compared to primary and newly added site server, we thought there could be some issue with memory in real however wanted to try other possibilities before going for any additional hardware/software. Server was not rebooted from long time, we gave it a try in order to release memory from operating system. we rebooted the server and waited for that night however replication server again crashed with same error message.
Now, we didn’t have any choice but to add more memory – we added 8GB of memory in addition to 16GB it already had and had to reboot the same post hardware change and were happy to think that replication will work smoothly now.
Same night replication server again went down as soon as EOD(End Of Day) started with exactly same error message, now our solution had two approach – first one to add more memory and see however that involved extra cost and also proving that we really need more memory and second point to upgrade our Replication server to see if there is any undocumented bug in this version.
We decided to go for the second approach and also to check how much memory is really consumed by replication server when it goes down, used below command to capture stats:
admin stats, “all”,”display”,”00:01:00″
We actually wanted to execute below command to gather memory statistics only:
admin stats, mem_detail_stats
However the above command was not supported in our installed replication server version and hence we had no choice but to go for the first one.
We added up all consumed memory and noticed that its not even reaching more than 5GB. We monitored the same for next two days, replication server kept failing with the same error message and memory consumption stayed within 5GB. This was the point when we went ahead and upgraded replication sever version in the same major.
Our current version was:
Replication Server/15.7.1/EBF 21863 SP120 rs1571sp120/Intel Itanium/HP-UX B.11.31/1/OPT64/Mon Dec 9 15:46:51 2013′
We upgraded to below version:
Replication Server/15.7.1/EBF 26747 SP306 rs1571sp306/Intel Itanium/HP-UX B.11.31/2237/OPT64/Tue May 16 18:23:25 2017′
Post upgrade, we monitored the replication server and also configured mem_detail_stats to gather memory statistics and replication server ran smoothly without any issue.
No more failure post upgrade of replication server. This seemed a undocumented bug in that version of HP-UX server for particular version of Sybase replication server for MSA.
Hope this helps to those who faces similar issue. Please leave your comment if you liked this post or have any feedback.