FAIL: process hdbindexserver HDB Indexserver not running

Hi all,

We have done HANA update to 69.3 and  i tried to start HANA  but Indexserver failed to start.Please suggest.

Error message is :FAIL: process hdbindexserver HDB Indexserver not running

Please find the trace below.I coudn't find any issues in indexserver_hostname.trc

[20507]{-1}[-1/-1] 2014-03-06 12:39:18.767254 i Basis            TraceStream.cpp(00385) : ==== Starting hdbindexserver, version 1.00.69.03.388114 (NewDB100_REL), build linuxx86_64 not set 2014-02-05 18:13:02 ld7270.wdf.sap.corp cc (SAP release 20130125, based on SUSE gcc43-4.3.4_20091019-0.22.17.5096.1.PTF.799300) 4.3.4 [gcc-4_3-branch revision 152973]

[20507]{-1}[-1/-1] 2014-03-06 12:39:18.767934 i Basis            TraceStream.cpp(00387) : MaxOpenFiles: 1048576

[20507]{-1}[-1/-1] 2014-03-06 12:39:18.767945 i Memory           MallocProxy.cpp(01226) : Installed malloc hooks

[20507]{-1}[-1/-1] 2014-03-06 12:39:18.767947 i Basis            ProcessorInfo.cpp(00392) : Using GDT segment limit to determine current CPU ID

[20507]{-1}[-1/-1] 2014-03-06 12:39:18.767948 i Basis            Timer.cpp(00583) : Using RDTSC for HR timer

[20507]{-1}[-1/-1] 2014-03-06 12:39:18.767949 i Memory           AllocatorImpl.cpp(00827) : Allocators activated

[20507]{-1}[-1/-1] 2014-03-06 12:39:18.767950 i Memory           AllocatorImpl.cpp(00843) : Using big block segment size 134217728

[20507]{-1}[-1/-1] 2014-03-06 12:39:18.767951 w Environment      Environment.cpp(00284) : Changing environment set IMSLERRPATH=/usr/sap/HP2/HDB51/exe//

[20507]{-1}[-1/-1] 2014-03-06 12:39:18.767952 w Environment      Environment.cpp(00284) : Changing environment set IMSLSERRPATH=/usr/sap/HP2/HDB51/exe//

[20507]{-1}[-1/-1] 2014-03-06 12:39:18.767953 w Environment      Environment.cpp(00284) : Changing environment set NODNSSAPTRANSHOST=1

[20507]{-1}[-1/-1] 2014-03-06 12:39:18.767954 w Environment      Environment.cpp(00263) : Changing environment set SAP_DIR_GLOBAL=/usr/sap/HP2/SYS/global

[20507]{-1}[-1/-1] 2014-03-06 12:39:18.780333 w Environment      Environment.cpp(00284) : Changing environment set SSL_WITH_OPENSSL=0

[20571]{-1}[-1/-1] 2014-03-06 12:39:19.595748 i assign           TREXIndexServer.cpp(00510) : assign to volume 2 started

[20571]{-1}[-1/-1] 2014-03-06 12:39:19.596471 i FileIO           FileStatistics.cpp(00247) : FileFactoryConfiguration::initial(path="/usr/sap/HP2/SYS/global/hdb/backint/", AsyncWriteSubmitActive=off,AsyncWriteSubmitBlocks=none,AsynReadSubmit=off,#SubmitQueues=1,#CompletionQueues=1)

[20571]{-1}[-1/-1] 2014-03-06 12:39:19.598388 i FileIO           FileStatistics.cpp(00247) : FileFactoryConfiguration::initial(path="/usr/sap/HP2/SYS/global/hdb/backint/", AsyncWriteSubmitActive=off,AsyncWriteSubmitBlocks=none,AsynReadSubmit=off,#SubmitQueues=1,#CompletionQueues=1)

[20571]{-1}[-1/-1] 2014-03-06 12:39:19.600409 i FileIO           FileStatistics.cpp(00247) : FileFactoryConfiguration::initial(path="/sapmnt/log/HP2/mnt00001/hdb00002/", AsyncWriteSubmitActive=off,AsyncWriteSubmitBlocks=none,AsynReadSubmit=off,#SubmitQueues=1,#CompletionQueues=1)

[20571]{-1}[-1/-1] 2014-03-06 12:39:19.602658 i FileIO           FileStatistics.cpp(00247) : FileFactoryConfiguration::initial(path="/sapmnt/data/HP2/mnt00001/hdb00002/", AsyncWriteSubmitActive=off,AsyncWriteSubmitBlocks=none,AsynReadSubmit=off,#SubmitQueues=1,#CompletionQueues=1)

[20571]{-1}[-1/-1] 2014-03-06 12:39:19.605645 i HistoryManager   HistoryManager.cpp(00450) : Maximum garbage collection parallelity set to 40

[20571]{-1}[-1/-1] 2014-03-06 12:39:19.605733 i PhysicalPageAcce DataVolumeImpl.cpp(00537) : Open volume file(s) in "/sapmnt/data/HP2/mnt00001/hdb00002/" (readOnly= 0, diagMode= 0)

[20571]{-1}[-1/-1] 2014-03-06 12:39:19.605945 i PhysicalPageAcce DataVolumeImpl.cpp(00595) : --opened volume file "datavolume_0000.dat"

[20571]{-1}[-1/-1] 2014-03-06 12:39:19.618108 i PersistenceManag PersistenceManagerImpl.cpp(03735) : Restart page version 1 loaded (96bytes)

[20571]{-1}[-1/-1] 2014-03-06 12:39:19.618135 i PersistenceManag PersistenceManagerImpl.cpp(03760) : Initial maximum known TID after restart: 331919412

[20571]{-1}[-1/-1] 2014-03-06 12:39:19.618138 i Logger           PersistenceManagerImpl.cpp(03768) : Newest known master commit position: 0xcb374790c

[20571]{-1}[-1/-1] 2014-03-06 12:39:19.618140 i Logger           PersistenceManagerImpl.cpp(03778) : Known last prepare commit position on volume 1: 0xd73cbc8

[20571]{-1}[-1/-1] 2014-03-06 12:39:19.618143 i Logger           PersistenceManagerImpl.cpp(03778) : Known last prepare commit position on volume 3: 0xa0c62082

[20571]{-1}[-1/-1] 2014-03-06 12:39:19.618145 i Logger           PersistenceManagerImpl.cpp(03778) : Known last prepare commit position on volume 4: 0x862002

[20571]{-1}[-1/-1] 2014-03-06 12:39:19.618146 i Logger           PersistenceManagerImpl.cpp(03778) : Known last prepare commit position on volume 5: 0x6d1d64882

[20571]{-1}[-1/-1] 2014-03-06 12:39:19.618148 i Logger           PersistenceManagerImpl.cpp(03778) : Known last prepare commit position on volume 6: 0x68c48cd42

[20571]{-1}[-1/-1] 2014-03-06 12:39:19.618149 i Logger           PersistenceManagerImpl.cpp(03778) : Known last prepare commit position on volume 7: 0x83adc2

[20571]{-1}[-1/-1] 2014-03-06 12:39:19.618570 i PersistenceManag PersistenceManagerImpl.cpp(03832) : Known DTX volume set [1,3,4,5,6,7], 2049 RTT entries

[20571]{-1}[-1/-1] 2014-03-06 12:39:21.063160 i PersistenceManag PersistenceSessionRegistry.cpp(00333) : Start loading open sessions and history cleanup files

[20571]{-1}[-1/-1] 2014-03-06 12:39:21.063393 i Logger           PersistenceSessionRegistry.cpp(00883) : Open session count at restart: 0/0, max known TID at restart: 331919412

[20571]{-1}[-1/-1] 2014-03-06 12:39:21.063405 i PersistenceManag PersistenceSessionRegistry.cpp(00340) : Loading 0 open session(s) and 0 history cleanup file(s) finished in 0.000223 seconds;

[20571]{-1}[-1/-1] 2014-03-06 12:39:21.354931 i Service_Startup  ContMgr.cc(00038) : Initializing system catalog.

[20571]{-1}[-1/-1] 2014-03-06 12:39:21.442614 i Service_Startup  ContMgr.cc(00122) : Initializing system catalog done.

[20571]{-1}[-1/-1] 2014-03-06 12:39:21.499337 e NameServer       TREXNameServer.cpp(05376) : getServer(SYSTEM:TTZD) without transaction

[20571]{-1}[-1/-1] 2014-03-06 12:39:21.534578 e NameServer       TREXNameServer.cpp(05376) : getServer(SYSTEM:TTZD) without transaction

[20654]{-1}[-1/-1] 2014-03-06 12:46:42.598473 i Service_Startup  CheckpointMgr.cc(01314) :  - RS segment loading progress: 10% done

[20654]{-1}[-1/-1] 2014-03-06 12:46:42.806225 i Service_Startup  CheckpointMgr.cc(00669) : 1106 RS segments loaded from Persistency in 441.10000000149012 sec (160.47154840118085MB/s)

[20571]{-1}[-1/-1] 2014-03-06 12:46:42.843374 i Service_Startup  MemSystem.cc(00144) : total memory size (MB) = 258329

[20571]{-1}[-1/-1] 2014-03-06 12:46:42.843480 i Logger           PersistenceManagerImpl.cpp(02038) : Preparing log recovery handler, start position 0xcb3747984

[20571]{-1}[-1/-1] 2014-03-06 12:46:42.843635 i Logger           LoggerImpl.cpp(00437) : Setting log segment backup timeout to 900s, original value was 0s

[20571]{-1}[-1/-1] 2014-03-06 12:46:42.903127 i PersistenceManag PersistenceManagerImpl.cpp(03241) : BACKUP DATA seems to have been executed, database should be recoverable from backup

[20571]{-1}[-1/-1] 2014-03-06 12:46:42.903153 i Logger           PersistenceManagerImpl.cpp(03443) : Deactivating REDO logging for log replay

[20571]{-1}[-1/-1] 2014-03-06 12:46:42.903164 i PersistenceManag PersistenceManagerImpl.cpp(03459) : Start garbage collection of history files

[20571]{-1}[-1/-1] 2014-03-06 12:46:42.903678 i PersistenceManag PersistenceManagerImpl.cpp(03469) : Garbage collection of history files finished: 0 cleanup files in 0.000488 seconds;

[20571]{-1}[-1/-1] 2014-03-06 12:46:42.903697 i Logger           PersistenceManagerImpl.cpp(03493) : Executing log replay in legacy mode, no savepoint log record written at 0xcb3747984 for savepoint 80613

[20571]{-1}[-1/-1] 2014-03-06 12:46:42.903792 i Logger           RecoveryHandlerImpl.cpp(00933) : RecoveryQueueCount used: 40

[20578]{-1}[-1/-1] 2014-03-06 12:46:48.476135 i Logger           RecoveryHandlerImpl.cpp(00373) : Stopping parsing of open segment LogSegment[0:0xcb3747940/Writing][GUID=EED01D07-6EF7-20140306-195257-37D0000001/PrevGUID=EED01D07-6EF7-20140305-022047-1CE0000003/PersGUID=EED01D07-6EF7-20140128-211915-2030000001/RestoreGUID=C87DF55B-0052-20130321-205051-29C0000001] at file position 20480, unused rest 1073721344B, termination reason: Invalid buffer header found at position 0x5000 in segment 0xcb3747940

[20571]{-1}[-1/-1] 2014-03-06 12:46:48.504658 i Logger           PersistenceManagerImpl.cpp(03553) : Starting log replay at position 0xcb3747984

[20571]{-1}[-1/-1] 2014-03-06 12:46:48.504682 i Logger           RecoveryHandlerImpl.cpp(01455) : Triggering recovery of remaining log

[20571]{-1}[-1/-1] 2014-03-06 12:46:48.504711 i Logger           RecoveryHandlerImpl.cpp(01503) : Finishing log recovery, waiting for in-process and in-load segments

[22160]{-1}[-1/-1] 2014-03-06 12:46:48.510859 i Logger           RecoveryHandlerImpl.cpp(01848) : Log recovery cannot expect any more data at position 0xcb3747a80, load states:

  - LogPartition[/sapmnt/log/HP2/mnt00001/hdb00002/:0]: #segments 0/1 processed, 1/1 loaded, 1 total, last loaded LogSegment[0:0xcb3747940/Writing][GUID=EED01D07-6EF7-20140306-195257-37D0000001/PrevGUID=EED01D07-6EF7-20140305-022047-1CE0000003/PersGUID=EED01D07-6EF7-20140128-211915-2030000001/RestoreGUID=C87DF55B-0052-20130321-205051-29C0000001]

[22160]{-1}[-1/-1] 2014-03-06 12:46:48.511075 i Logger           RecoveryHandlerImpl.cpp(02147) : Recovery finished at log position 0xcb3747a80

[22143]{-1}[-1/-1] 2014-03-06 12:46:48.512590 i Logger           RecoveryHandlerImpl.cpp(03324) : Signaling finish wait barrier

[20571]{-1}[-1/-1] 2014-03-06 12:46:48.514053 i Logger           LogSegment.cpp(00566) : Closing open log segments after log recovery completed at position 0xcb3747a80

[20571]{-1}[-1/-1] 2014-03-06 12:46:48.514067 i Logger           LogSegment.cpp(00649) : Closing open on-disk segment LogSegment[0/0:0xcb3747940/GUID=EED01D07-6EF7-20140306-195257-37D0000001/PrevGUID=EED01D07-6EF7-20140305-022047-1CE0000003,TS=2014-03-06 11:52:57.893645/Writing/0x0]@0x00007f3ed5db1180 after log recovery completed at position 0xcb3747a80

[20571]{-1}[-1/-1] 2014-03-06 12:46:48.516305 i Logger           LoggerImpl.cpp(00920) : Replayed 16128B (0MB) of log in 5.6126 seconds; 0.00274042MB/s; max known TID=331921464

[20571]{-1}[-1/-1] 2014-03-06 12:46:48.516453 i Logger           LoggerImpl.cpp(00710) : Starting logger with 8 log buffers per partition, 1024KB each, segment size 1024MB, log mode normal

[20571]{-1}[-1/-1] 2014-03-06 12:46:48.540389 i Logger           RecoveryHandlerImpl.cpp(01749) : Termination of 0 indoubt transactions was executed in 0.010026 seconds;

[20571]{-1}[-1/-1] 2014-03-06 12:46:48.540844 i Logger           PersistenceManagerImpl.cpp(03661) : Activating REDO logging after log replay

[20571]{-1}[-1/-1] 2014-03-06 12:46:48.542066 i LogReplay        RowStoreTransactionCallback.cc(00226) : starting master-slave DTX consistency check

[20571]{-1}[-1/-1] 2014-03-06 12:46:49.546627 i LogReplay        RowStoreTransactionCallback.cc(00301) : finished master-slave DTX consistency check

[20583]{-1}[-1/-1] 2014-03-06 12:46:50.234499 i PersistenceManag VirtualFileStatsProxy.cpp(00226) : VirtualFile LOB statistics initialized: 0 CD entries in 165.00 usec

[20584]{-1}[-1/-1] 2014-03-06 12:46:50.882755 i PersistenceManag VirtualFileStatsProxy.cpp(00166) : VirtualFile statistics initialized: 149128/149128 CD entries in 648.41 msec

[20571]{-1}[-1/-1] 2014-03-06 12:46:50.949235 i PersistenceManag PersistenceManagerImpl.cpp(02127) : Activating periodic savepoint, frequency 300

[20571]{-1}[-1/-1] 2014-03-06 12:46:50.957255 i Service_Startup  transmgmt.cc(05357) : Starting up global transaction coordinator

[22162]{-1}[-1/-1] 2014-03-06 12:46:50.957535 i FileIO           FileStatistics.cpp(00247) : FileFactoryConfiguration::initial(path="/usr/sap/HP2/HDB51/backup/log/", AsyncWriteSubmitActive=off,AsyncWriteSubmitBlocks=none,AsynReadSubmit=off,#SubmitQueues=1,#CompletionQueues=1)

[22162]{-1}[-1/-1] 2014-03-06 12:46:50.983437 i FileIO           FileStatistics.cpp(00247) : FileFactoryConfiguration::initial(path="/usr/sap/HP2/HDB51/backup/log/", AsyncWriteSubmitActive=off,AsyncWriteSubmitBlocks=none,AsynReadSubmit=off,#SubmitQueues=1,#CompletionQueues=1)

[20571]{-1}[-1/-1] 2014-03-06 12:46:51.330089 i Service_Startup  mm_recovery.cc(01283) : Rebuilding page lists.

[20571]{-1}[-1/-1] 2014-03-06 12:46:54.631667 i Service_Startup  mm_recovery.cc(00854) : RS: metadata & data are separated

[20571]{-1}[-1/-1] 2014-03-06 12:46:54.642214 i Service_Startup  ptime_master_start.cc(00715) : Rebuilding system indexes.

[20571]{-1}[-1/-1] 2014-03-06 12:46:54.999763 i Service_Startup  ptime_master_start.cc(00721) : Rebuilding system indexes successfully done.

[20571]{-1}[-1/-1] 2014-03-06 12:46:54.999792 i Service_Startup  ptime_master_start.cc(00723) : Rebuilding indexes.

[20571]{-1}[-1/-1] 2014-03-06 12:46:55.014895 i Service_Startup  IndexManager_rebuild.cc(00305) : Number of indexes: 2791

[20571]{-1}[-1/-1] 2014-03-06 12:46:55.014910 i Service_Startup  IndexManager_rebuild.cc(00327) : Number of large indexes: 12

Thanks,

Raghu