- Remove From My Forums
-
Question
-
Good day,
Last week, I started receiving this error inconjunction with an SSIS package which executes at a particular time. Any help would be appreciated. Here are the details
Environment: Virtualized Server
Sql Version: Microsoft SQL Server 2005 — 9.00.4053.00 (Intel X86)(Build 3790: Service Pack 2)
SSIS Package: Copies a file based database from server A to SQL Server for importing into SQL Server database.
SSIS Executed: Every two hours
Other Components: ACCDB which contains links to the file-based database, using Topspeed Drivers; SQL Server contains a linked object to the ACCDB.Symptoms: Throughout the day, this process runs. When it is executed at 6:00 pm, the server crashes. I attempted to change the time that it runs in case some other process was causing resource to not be availble, and looked at the AV scan
settings and schedule. Nothing has helped. This issue has been occurring for the past week. I’m not sure how to interpret the information in the error log or what to try next. Your input is appreciated.The error log is below:
2012-04-13 18:45:50.53 Server Error: 17311, Severity: 16, State: 1.
2012-04-13 18:45:50.53 Server SQL Server is terminating because of fatal exception c0000005. This error may be caused by an unhandled Win32 or C++ exception, or by an access violation encountered during exception handling. Check
the SQL error log for any related stack dumps or messages. This exception forces SQL Server to shutdown. To recover from this error, restart the server (unless SQLAgent is configured to auto restart).
2012-04-13 18:45:50.68 Server Using ‘dbghelp.dll’ version ‘4.0.5’
2012-04-13 18:45:50.72 Server **Dump thread — spid = 0, PSS = 0x00000000, EC = 0x00000000
2012-04-13 18:45:50.72 Server ***Stack Dump being sent to C:Program FilesMicrosoft SQL ServerMSSQL.1MSSQLLOGSQLDump0005.txt
2012-04-13 18:45:50.72 Server * *******************************************************************************
2012-04-13 18:45:50.72 Server *
2012-04-13 18:45:50.72 Server * BEGIN STACK DUMP:
2012-04-13 18:45:50.72 Server * 04/13/12 18:45:50 spid 0
2012-04-13 18:45:50.72 Server *
2012-04-13 18:45:50.72 Server * ex_handle_except encountered exception C0000005 — Server terminating
2012-04-13 18:45:50.72 Server *
2012-04-13 18:45:50.72 Server *
2012-04-13 18:45:50.72 Server * MODULE BASE
END SIZE
2012-04-13 18:45:50.72 Server * sqlservr 01000000 02C1FFFF 01c20000
2012-04-13 18:45:50.72 Server * ntdll 7C800000 7C8C2FFF 000c3000
2012-04-13 18:45:50.72 Server * kernel32 77E40000 77F41FFF 00102000
2012-04-13 18:45:50.72 Server * MSVCR80 78130000 781CAFFF 0009b000
2012-04-13 18:45:50.72 Server * msvcrt 77BA0000 77BF9FFF 0005a000
2012-04-13 18:45:50.72 Server * MSVCP80 7C420000 7C4A6FFF 00087000
2012-04-13 18:45:50.72 Server * ADVAPI32 7D1E0000 7D27BFFF 0009c000
2012-04-13 18:45:50.72 Server * RPCRT4 77C50000 77CEFFFF 000a0000
2012-04-13 18:45:50.72 Server * Secur32 76F50000 76F62FFF 00013000
2012-04-13 18:45:50.72 Server * sqlos 344D0000 344D4FFF 00005000
2012-04-13 18:45:50.72 Server * USERENV 76920000 769E1FFF 000c2000
2012-04-13 18:45:50.72 Server * USER32 77380000 77410FFF 00091000
2012-04-13 18:45:50.72 Server * GDI32 77C00000 77C48FFF 00049000
2012-04-13 18:45:50.72 Server * WINMM 76AA0000 76ACCFFF 0002d000
2012-04-13 18:45:50.72 Server * opends60 333E0000 333E6FFF 00007000
2012-04-13 18:45:50.72 Server * NETAPI32 71C40000 71C96FFF 00057000
2012-04-13 18:45:50.72 Server * psapi 76B70000 76B7AFFF 0000b000
2012-04-13 18:45:50.72 Server * instapi 48060000 48069FFF 0000a000
2012-04-13 18:45:50.72 Server * sqlevn70 4F610000 4F7B8FFF 001a9000
2012-04-13 18:45:50.72 Server * NTMARTA 77E00000 77E20FFF 00021000
2012-04-13 18:45:50.72 Server * WLDAP32 76F10000 76F3DFFF 0002e000
2012-04-13 18:45:50.72 Server * SAMLIB 7E020000 7E02EFFF 0000f000
2012-04-13 18:45:50.73 Server * ole32 77670000 777A8FFF 00139000
2012-04-13 18:45:50.73 Server * rsaenh 68000000 68034FFF 00035000
2012-04-13 18:45:50.73 Server * AUTHZ 76C40000 76C53FFF 00014000
2012-04-13 18:45:50.73 Server * MSCOREE 33DF0000 33E39FFF 0004a000
2012-04-13 18:45:50.73 Server * msv1_0 76C90000 76CB6FFF 00027000
2012-04-13 18:45:50.73 Server * cryptdll 34050000 3405BFFF 0000c000
2012-04-13 18:45:50.73 Server * WS2_32 71C00000 71C16FFF 00017000
2012-04-13 18:45:50.73 Server * WS2HELP 71BF0000 71BF7FFF 00008000
2012-04-13 18:45:50.73 Server * iphlpapi 76CF0000 76D09FFF 0001a000
2012-04-13 18:45:50.73 Server * kerberos 340B0000 34108FFF 00059000
2012-04-13 18:45:50.73 Server * MSASN1 34110000 34121FFF 00012000
2012-04-13 18:45:50.73 Server * schannel 34140000 34168FFF 00029000
2012-04-13 18:45:50.73 Server * CRYPT32 34170000 34202FFF 00093000
2012-04-13 18:45:50.73 Server * COMRES 77010000 770D5FFF 000c6000
2012-04-13 18:45:50.73 Server * XOLEHLP 34250000 34255FFF 00006000
2012-04-13 18:45:50.73 Server * MSDTCPRX 34260000 342D8FFF 00079000
2012-04-13 18:45:50.73 Server * OLEAUT32 7D0E0000 7D16AFFF 0008b000
2012-04-13 18:45:50.73 Server * msvcp60 342E0000 34344FFF 00065000
2012-04-13 18:45:50.73 Server * MTXCLU 34350000 34368FFF 00019000
2012-04-13 18:45:50.73 Server * VERSION 77B90000 77B97FFF 00008000
2012-04-13 18:45:50.73 Server * WSOCK32 71BB0000 71BB8FFF 00009000
2012-04-13 18:45:50.73 Server * CLUSAPI 34370000 34381FFF 00012000
2012-04-13 18:45:50.73 Server * RESUTILS 34390000 343A2FFF 00013000
2012-04-13 18:45:50.73 Server * mswsock 71B20000 71B60FFF 00041000
2012-04-13 18:45:50.73 Server * DNSAPI 76ED0000 76EF9FFF 0002a000
2012-04-13 18:45:50.73 Server * winrnr 76F70000 76F76FFF 00007000
2012-04-13 18:45:50.73 Server * rasadhlp 76F80000 76F84FFF 00005000
2012-04-13 18:45:50.73 Server * security 62740000 62743FFF 00004000
2012-04-13 18:45:50.73 Server * msfte 62E10000 63068FFF 00259000
2012-04-13 18:45:50.73 Server * dbghelp 63080000 63194FFF 00115000
2012-04-13 18:45:50.73 Server * WINTRUST 76BB0000 76BDBFFF 0002c000
2012-04-13 18:45:50.73 Server * imagehlp 76C10000 76C37FFF 00028000
2012-04-13 18:45:50.73 Server * dssenh 68100000 68126FFF 00027000
2012-04-13 18:45:50.73 Server * hnetcfg 63460000 634B9FFF 0005a000
2012-04-13 18:45:50.73 Server * wshtcpip 71AE0000 71AE7FFF 00008000
2012-04-13 18:45:50.73 Server * ntdsapi 63580000 63593FFF 00014000
2012-04-13 18:45:50.73 Server * xpsp2res 635A0000 63864FFF 002c5000
2012-04-13 18:45:50.73 Server * CLBCatQ 777B0000 77832FFF 00083000
2012-04-13 18:45:50.73 Server * sqlncli 63870000 63A96FFF 00227000
2012-04-13 18:45:50.73 Server * COMCTL32 77530000 775C6FFF 00097000
2012-04-13 18:45:50.73 Server * comdlg32 00770000 007B8FFF 00049000
2012-04-13 18:45:50.73 Server * SHLWAPI 7D180000 7D1D1FFF 00052000
2012-04-13 18:45:50.73 Server * SHELL32 7C8D0000 7D0CEFFF 007ff000
2012-04-13 18:45:50.73 Server * comctl32 77420000 77522FFF 00103000
2012-04-13 18:45:50.73 Server * SQLNCLIR 63AD0000 63B02FFF 00033000
2012-04-13 18:45:50.73 Server * msftepxy 63C90000 63CA4FFF 00015000
2012-04-13 18:45:50.73 Server * xpsqlbot 64120000 64125FFF 00006000
2012-04-13 18:45:50.73 Server * xpstar90 64140000 6418AFFF 0004b000
2012-04-13 18:45:50.73 Server * SQLSCM90 641A0000 641A8FFF 00009000
2012-04-13 18:45:50.73 Server * ODBC32 641E0000 6421DFFF 0003e000
2012-04-13 18:45:50.73 Server * BatchParser90 64220000 6423EFFF 0001f000
2012-04-13 18:45:50.73 Server * ATL80 7C630000 7C64AFFF 0001b000
2012-04-13 18:45:50.73 Server * odbcint 64290000 642A6FFF 00017000
2012-04-13 18:45:50.73 Server * xpstar90 64440000 64465FFF 00026000
2012-04-13 18:45:50.73 Server * xplog70 64470000 6447BFFF 0000c000
2012-04-13 18:45:50.73 Server * xplog70 64490000 64492FFF 00003000
2012-04-13 18:45:50.73 Server * oledb32 64730000 647A8FFF 00079000
2012-04-13 18:45:50.73 Server * MSDART 63C20000 63C39FFF 0001a000
2012-04-13 18:45:50.73 Server * OLEDB32R 64AF0000 64B00FFF 00011000
2012-04-13 18:45:50.73 Server * comsvcs 64B10000 64C54FFF 00145000
2012-04-13 18:45:50.73 Server * ACEOLEDB 64C60000 64CBEFFF 0005f000
2012-04-13 18:45:50.73 Server * MSVCR90 64CC0000 64D62FFF 000a3000
2012-04-13 18:45:50.73 Server * mso 64D80000 65F5FFFF
011e0000
2012-04-13 18:45:50.73 Server * msi 65F70000 6622DFFF
002be000
2012-04-13 18:45:50.73 Server * ACECORE 663B0000 665C9FFF 0021a000
2012-04-13 18:45:50.73 Server * MSVCP90 665E0000 6666DFFF 0008e000
2012-04-13 18:45:50.73 Server * ACEWSTR 66AD0000 66BA1FFF 000d2000
2012-04-13 18:45:50.73 Server * MSORES 68130000 6C659FFF 0452a000
2012-04-13 18:45:50.73 Server * MSOINTL 66BD0000 66E31FFF 00262000
2012-04-13 18:45:50.73 Server * ACEES 67370000 67419FFF 000aa000
2012-04-13 18:45:50.73 Server * VBAJET32 67430000 6743DFFF 0000e000
2012-04-13 18:45:50.73 Server * expsrv 67450000 674C0FFF 00071000
2012-04-13 18:45:50.73 Server * sqlvdi 67030000 67051FFF 00022000
2012-04-13 18:45:50.73 Server * office 70660000 70A6EFFF 0040f000
2012-04-13 18:45:50.73 Server * UxTheme 71B70000 71BA5FFF 00036000
2012-04-13 18:45:50.73 Server * dbghelp 70B70000 70C84FFF 00115000
2012-04-13 18:45:50.73 Server *
2012-04-13 18:45:50.73 Server * Edi: 67FBE8E4: 001B7068 00000000 00000000 00000000 00000000 00000000
2012-04-13 18:45:50.73 Server * Esi: 00000000:
2012-04-13 18:45:50.73 Server * Eax: 67FBE7C8: 000042AC 00000000 00000000 77E4BEF7 00000000 00000002
2012-04-13 18:45:50.73 Server * Ebx: 0000003F:
2012-04-13 18:45:50.73 Server * Ecx: 67FBEE38: 00000000 00010007 00000000 0090008E 001B706C 00000000
2012-04-13 18:45:50.73 Server * Edx: 0000003D:
2012-04-13 18:45:50.73 Server * Eip: 77E4BEF7: 10C2C95E 01093D00 840FC000 00000095 F29DE950 9090FFFF
2012-04-13 18:45:50.73 Server * Ebp: 67FBE818: 67FBE85C 021AE24C 000042AC 00000000 00000000 00000000
2012-04-13 18:45:50.73 Server * SegCs: 7813001B: 00000000 00000000 00000000 00000000 00000000 00000000
2012-04-13 18:45:50.73 Server * EFlags: 00000246:
2012-04-13 18:45:50.73 Server * Esp: 67FBE7C4: 00000000 000042AC 00000000 00000000 77E4BEF7 00000000
2012-04-13 18:45:50.73 Server * SegSs: 78130023: 00000000 00000000 00000000 00000000 00000000 00000000
2012-04-13 18:45:50.74 Server * *******************************************************************************
2012-04-13 18:45:50.74 Server * ——————————————————————————-
2012-04-13 18:45:50.74 Server * Short Stack Dump
2012-04-13 18:45:50.75 Server 77E4BEF7 Module(kernel32+0000BEF7)
2012-04-13 18:45:50.75 Server 021AE24C Module(sqlservr+011AE24C)
2012-04-13 18:45:50.75 Server 021B2160 Module(sqlservr+011B2160)
2012-04-13 18:45:50.75 Server 021B1D0D Module(sqlservr+011B1D0D)
2012-04-13 18:45:50.75 Server 02422369 Module(sqlservr+01422369)
2012-04-13 18:45:50.75 Server 77E761B7 Module(kernel32+000361B7)
2012-04-13 18:45:50.75 Server 77E792A3 Module(kernel32+000392A3)
2012-04-13 18:45:50.77 Server Stack Signature for the dump is 0x1A38D65C
2012-04-13 18:45:53.94 Server External dump process return code 0x20000001.
External dump process returned no errors.Thank you.
Answers
-
Hi GFrank,
A similar error is reported in
this KB article, with the fix initially released in Cumulative Update 9 for SQL Server 2005 Service Pack 3. I would recommend you apply the latest Service Pack via
SQL Server 2005 to avoid known issues.
Stephanie Lv
TechNet Community Support
-
Marked as answer by
Tuesday, April 24, 2012 8:36 AM
-
Marked as answer by
Case 1, with Dump generated
Issue definition:
========
Customer have SQL 2008/2012/2014 running on the same server, and then applied the SP4 for SQL Server 2012 in Windows update. But after this customer saw some issue from application side, then you rollback the update.
And then Customer applied the SP3 for SQL Server 2012, but after that the SQL Server service can’t startup with following message and dump generated. Customer also tried to repair SQL Server many times, it doesn’t work.
2018-02-22 18:31:43.52 Server Error: 17311, Severity: 16, State: 1.
2018-02-22 18:31:43.52 Server SQL Server is terminating because of fatal exception c06d007e. This error may be caused by an unhandled Win32 or C++ exception, or by an access violation encountered during exception handling. Check the SQL error log for any related stack dumps or messages. This exception forces SQL Server to shutdown. To recover from this error, restart the server (unless SQLAgent is configured to auto restart).
2018-02-22 18:31:43.53 Server Using ‘dbghelp.dll’ version ‘4.0.5’
2018-02-22 18:31:43.53 Server **Dump thread — spid = 0, EC = 0x0000000000000000
2018-02-22 18:31:43.53 Server ***Stack Dump being sent to E:Program FilesMicrosoft SQL ServerMSSQL11.MSSQLSERVERMSSQLLOGSQLDump0094.txt
2018-02-22 18:31:43.53 Server * *******************************************************************************
2018-02-22 18:31:43.53 Server *
2018-02-22 18:31:43.53 Server * BEGIN STACK DUMP:
2018-02-22 18:31:43.53 Server * 02/22/18 18:31:43 spid 11352
2018-02-22 18:31:43.53 Server *
2018-02-22 18:31:43.53 Server * ex_handle_except encountered exception C06D007E — Server terminating
Issue analysis and troubleshooting thinking
========
-
In the live meeting with customer, we first checking the setup log. And we saw that the SP4 has been applied, but after a while there is another setup folder generated reporting the failed error message:
Exception type: Microsoft.SqlServer.Chainer.Infrastructure.ChainerInvalidOperationException
Message:
The NT service ‘SQLSERVERAGENT’ could not be started.
HResult : 0x84b20001
FacilityCode : 1202 (4b2)
ErrorCode : 1 (0001)
Data:
HelpLink.EvtType = 0xAE7B8BF3@0xF762992B@1202@1
DisableWatson = true
Stack:
at Microsoft.SqlServer.Configuration.SetupExtension.RunPatchAllInstanceAction.RestoreServiceStates()
at Microsoft.SqlServer.Configuration.SetupExtension.RunPatchAllInstanceAction.ExecuteAction(String actionId)
at Microsoft.SqlServer.Chainer.Infrastructure.Action.Execute(String actionId, TextWriter errorStream)
at Microsoft.SqlServer.Setup.Chainer.Workflow.ActionInvocation.ExecuteActionHelper(TextWriter statusStream, ISequencedAction actionToRun, ServiceContainer context)
- Then we get the dump file from customer and performed a detailed analysis . This callstack seems that we have issue while Initialize Server Components(InitServerComponents), and then we get the detailed component name “SqlServerSpatial110.dll”
# Child-SP RetAddr Call Site
00 00000000`01bf9cf8 000007fe`fd3810ac ntdll!ZwWaitForSingleObject+0xa
01 00000000`01bf9d00 00000000`ff6af3ee KERNELBASE!WaitForSingleObjectEx+0x79
02 00000000`01bf9da0 00000000`ff6af874 sqlservr!CDmpDump::DumpInternal+0x20e
03 00000000`01bf9e40 00000000`ff6af919 sqlservr!CDmpDump::DumpFilter+0x64
04 00000000`01bf9f10 00000000`74d712e3 sqlservr!`CDmpDump::Dump’::`1′::filt$0+0x19
05 00000000`01bf9f40 00000000`7769bf9d msvcr100!__C_specific_handler+0x97
06 00000000`01bf9fb0 00000000`776704ca ntdll!RtlpExecuteHandlerForException+0xd
07 00000000`01bf9fe0 00000000`77673c1f ntdll!RtlDispatchException+0x45a
08 00000000`01bfa6c0 000007fe`fd39a06d ntdll!RtlRaiseException+0x22f
09 00000000`01bfb070 00000000`ff6af8ec KERNELBASE!RaiseException+0x39
0a 00000000`01bfb140 000007fe`cd6bd4c4 sqlservr!CDmpDump::Dump+0x4c
0b 00000000`01bfb180 000007fe`cd6bdf06 sqllang!SQLDumperLibraryInvoke+0x2e4
0c 00000000`01bfb420 000007fe`cd684b6b sqllang!CImageHelper::DoMiniDump+0x426
0d 00000000`01bfb610 000007fe`cd683f6a sqllang!stackTrace+0xbdb
0e 00000000`01bfd050 000007fe`e0312a75 sqllang!stackTraceCallBack+0xca
0f 00000000`01bfd090 00000000`775abc20 sqldk!ex_handle_except+0x125
10 00000000`01bfd2e0 00000000`776a9035 kernel32!UnhandledExceptionFilter+0x160
11 00000000`01bfd3c0 00000000`77687398 ntdll!RtlUserThreadStart$filt$0+0x45
12 00000000`01bfd3f0 00000000`7769bf9d ntdll!__C_specific_handler+0x8c
13 00000000`01bfd460 00000000`776704ca ntdll!RtlpExecuteHandlerForException+0xd
14 00000000`01bfd490 00000000`77673c1f ntdll!RtlDispatchException+0x45a
15 00000000`01bfdb70 000007fe`fd39a06d ntdll!RtlRaiseException+0x22f
16 00000000`01bfe520 000007fe`ccb7bc0b KERNELBASE!RaiseException+0x39
17 00000000`01bfe5f0 000007fe`cdb8e40f sqllang!__delayLoadHelper2+0x199
18 00000000`01bfe6a0 000007fe`cdb49a75 sqllang!_tailMerge_SqlServerSpatial110_dll+0x3f
19 (Inline Function) ———`——— sqllang!InitializeSpatialNativeImpl+0x26
1a 00000000`01bfe710 00000000`ff69af2a sqllang!SQLLangInitComponents+0x275
1b 00000000`01bfe770 00000000`ff69c094 sqlservr!InitServerComponents+0x8a
1c 00000000`01bfe7d0 00000000`ff693c48 sqlservr!sqlservr_main+0x3b4
1d 00000000`01bfea20 000007fe`fe32a82d sqlservr!sqlservr_main_thread+0x178
1e 00000000`01bffd30 00000000`775259cd sechost!ScSvcctrlThreadA+0x25
1f 00000000`01bffd60 00000000`7768383d kernel32!BaseThreadInitThunk+0xd
20 00000000`01bffd90 00000000`00000000 ntdll!RtlUserThreadStart+0x1d
Looked at the DLL being loaded
szDll : 0x7fecdeef9c0 : «SqlServerSpatial110.dll» [Type: char *]
-
Then we checked the SqlServerSpatial110.dll both in C:WindowsSystem32 and C:WindowsSystemWOW64.
System32
WOW64:
-
And then we checked this SqlServerSpatial110.dll properties in another working server as below. Then we got the root cause that the current server SqlServerSpatial110.dll original filename is SqlServerSpatial120.dll(it is SQL 2014). But in this case, somehow is has been changed, but we don’t know why, we only get to the result.
-
Then we tried to copy the correct one from another server to current server(system32 and systemWOW64 respectively). But this doesn’t work. After we restart the SQL Server it failed again, with same error.
-
Then we come up with another solution, we can rename the current SqlServerSpatial110.dll to SqlServerSpatial110.dll.old and then launch the repair again. This worked.
-
After repairing, we can see the new SqlServerSpatial110.dll has been created automatically with correct SP3 version(same with the current SQL Server version)
Root cause:
========
Somehow, SqlServerSpatial110.dll has been changed.
Resolution:
========
-
Rename the SqlServerSpatial110.dll under C:WindowsSystem32 and C:WindowsSystemWOW64
-
Launch the repair for SQL Server 2012. This resolved this issue successfully.
Case 2, there is no dump generated
Issue definition:
========
After you patch the SQL Server 2016, 9 instances failed to start due to the following error.
2020-02-22 18:43:17.14 Server Error: 17311, Severity: 16, State: 1.
2020-02-22 18:43:17.14 Server SQL Server is terminating because of fatal exception c06d007e. This error may be caused by an unhandled Win32 or C++ exception, or by an access violation encountered during exception handling. Check the SQL error log for any related stack dumps or messages. This exception forces SQL Server to shutdown. To recover from this error, restart the server (unless SQLAgent is configured to auto restart).
2020-02-22 18:43:17.14 Server Using ‘dbghelp.dll’ version ‘4.0.5’
2020-02-22 18:43:17.14 Server **Dump thread — spid = 0, EC = 0x0000000000000000
2020-02-22 18:43:17.14 Server * *******************************************************************************
2020-02-22 18:43:17.14 Server *
2020-02-22 18:43:17.14 Server * BEGIN STACK DUMP:
2020-02-22 18:43:17.14 Server * 02/22/20 18:43:17 spid 8832
2020-02-22 18:43:17.14 Server *
2020-02-22 18:43:17.14 Server * ex_handle_except encountered exception C06D007E — Server terminating
2020-02-22 18:43:17.22 Server Stack Signature for the dump is 0x00000001A124A749
2020-02-22 18:43:17.22 Server Unable to create dump because SQLDUMPER library is not available.
2020-02-22 18:43:17.22 Server SQL Server shutdown has been initiated
Issue analysis:
========
-
Based on the log, this issue should happened when loading some DLL.
-
But as there no dump generated due to Unable to create dump because SQLDUMPER library is not available(from the following analysis, we can also see that dbgheko.dll not loaded yet).
-
So we captured a process monitor, and then analyze the process monitor log.
-
From the process monitor log, we can’t get the callstack which can help us to identify the DLL.
-
So we are not able to find the DLL from the callstack, but we can compare the DLL which has already been loaded with a working server, and find out all the non-loaded DLL, and maybe we can find more clue.
-
Compared the working server loaded DLL and this problematic server. All the yellow-mark DLL are non-loaded DLL in the problematic SQL.
Working Server:
<<<<<<<<<<<<<<<
SQLNCLIRDAR11.RLL 0x18261ee0000 0x38000 c:Program FilesMicrosoft SQL Server140Shared1033SQLNCLIRDAR11.RLL Microsoft Corporation 2011.0110.5069.066 ((SQL11_OLEDB_AAD).160321-0812 ) 3/21/2016 11:56:15 PM
instapi140.dll 0x7ffc21c80000 0x13000 c:Program FilesMicrosoft SQL Server140Sharedinstapi140.dll Microsoft Corporation 2017.0140.3238.01 ((SQLServer2017-CU14).190913-2228) 9/14/2019 6:36:59 AM
sqlnclirda11.dll 0x59e10000 0x359000 c:Program FilesMicrosoft SQL Server140Sharedsqlnclirda11.dll Microsoft Corporation 2011.0110.5069.066 ((SQL11_OLEDB_AAD).160321-0812 ) 3/21/2016 11:57:40 PM
BatchParser.dll 0x7ffbf7990000 0x2c000 C:Program FilesMicrosoft SQL ServerMSSQL14.MSSQLSERVERMSSQLBinnBatchParser.dll Microsoft Corporation 2017.0140.3238.01 ((SQLServer2017-CU14).190913-2228) 9/14/2019 6:36:47 AM
dbghelp.dll 0x59c70000 0x196000 C:Program FilesMicrosoft SQL ServerMSSQL14.MSSQLSERVERMSSQLBinndbghelp.dll Microsoft Corporation 6.12.0002.633 (debuggers(dbg).100201-1211) 2/2/2010 4:15:44 AM
ftimport.dll 0x60000000 0x25000 c:Program FilesMicrosoft SQL ServerMSSQL14.MSSQLSERVERMSSQLBinnftimport.dll Microsoft Corporation 12.0.7031.9999 11/1/2006 12:27:20 PM
hkcompile.dll 0x7ffbf86f0000 0x162000 C:Program FilesMicrosoft SQL ServerMSSQL14.MSSQLSERVERMSSQLBinnhkcompile.dll Microsoft Corporation 2017.0140.3238.01 ((SQLServer2017-CU14).190913-2228) 9/14/2019 6:47:36 AM
hkengine.dll 0x7ffbf7fe0000 0x70b000 C:Program FilesMicrosoft SQL ServerMSSQL14.MSSQLSERVERMSSQLBinnhkengine.dll Microsoft Corporation 2017.0140.3238.01 ((SQLServer2017-CU14).190913-2228) 9/14/2019 6:51:33 AM
hkruntime.dll 0x7ffbf8860000 0x2d1000 C:Program FilesMicrosoft SQL ServerMSSQL14.MSSQLSERVERMSSQLBinnhkruntime.dll Microsoft Corporation 2017.0140.3238.01 ((SQLServer2017-CU14).190913-2228) 9/14/2019 6:52:55 AM
MSFTE.DLL 0x49980000 0x3ae000 C:Program FilesMicrosoft SQL ServerMSSQL14.MSSQLSERVERMSSQLBinnMSFTE.DLL Microsoft Corporation 12.0.6828.0 8/28/2006 7:15:55 PM
opends60.dll 0x7ffbf8cb0000 0x9000 C:Program FilesMicrosoft SQL ServerMSSQL14.MSSQLSERVERMSSQLBinnopends60.dll Microsoft Corporation 2017.0140.3238.01 ((SQLServer2017-CU14).190913-2228) 9/14/2019 6:36:08 AM
qds.dll 0x7ffbf8b80000 0x12e000 C:Program FilesMicrosoft SQL ServerMSSQL14.MSSQLSERVERMSSQLBinnqds.dll Microsoft Corporation 2017.0140.3238.01 ((SQLServer2017-CU14).190913-2228) 9/14/2019 9:38:39 AM
sqlevn70.rll 0x17cc64a0000 0x310000 c:Program FilesMicrosoft SQL ServerMSSQL14.MSSQLSERVERMSSQLBinnResources1033sqlevn70.rll Microsoft Corporation 2017.0140.3238.01 ((SQLServer2017-CU14).190913-2228) 9/14/2019 6:36:13 AM
xplog70.RLL 0x183ea3d0000 0x4000 C:Program FilesMicrosoft SQL ServerMSSQL14.MSSQLSERVERMSSQLBinnResources1033xplog70.RLL Microsoft Corporation 2017.0140.1000.169 ((SQLServer).170822-2340) 8/23/2017 7:50:34 AM
xpstar.rll 0x183e93a0000 0xd000 c:Program FilesMicrosoft SQL ServerMSSQL14.MSSQLSERVERMSSQLBinnResources1033xpstar.rll Microsoft Corporation 2017.0140.1000.169 ((SQLServer).170822-2340) 8/23/2017 7:50:29 AM
secforwarder.dll 0x7ffbf8b40000 0xa000 C:Program FilesMicrosoft SQL ServerMSSQL14.MSSQLSERVERMSSQLBinnsecforwarder.dll Microsoft Corporation 2017.0140.3238.01 ((SQLServer2017-CU14).190913-2228) 9/14/2019 6:45:38 AM
SqlAccess.dll 0x7ffbf7f60000 0x75000 C:Program FilesMicrosoft SQL ServerMSSQL14.MSSQLSERVERMSSQLBinnSqlAccess.dll Microsoft Corporation 2017.0140.3238.01 ((SQLServer2017-CU14).190913-2228) 9/14/2019 6:37:39 AM
sqldk.dll 0x7ffbf8cc0000 0x4fc000 C:Program FilesMicrosoft SQL ServerMSSQL14.MSSQLSERVERMSSQLBinnsqldk.dll Microsoft Corporation 2017.0140.3238.01 ((SQLServer2017-CU14).190913-2228) 9/14/2019 9:35:28 AM
sqllang.dll 0x7ffbf9af0000 0x2782000 C:Program FilesMicrosoft SQL ServerMSSQL14.MSSQLSERVERMSSQLBinnsqllang.dll Microsoft Corporation 2017.0140.3238.01 ((SQLServer2017-CU14).190913-2228) 9/14/2019 9:44:02 AM
sqlmin.dll 0x7ffbfc280000 0x31fe000 C:Program FilesMicrosoft SQL ServerMSSQL14.MSSQLSERVERMSSQLBinnsqlmin.dll Microsoft Corporation 2017.0140.3238.01 ((SQLServer2017-CU14).190913-2228) 9/14/2019 9:38:05 AM
SQLOS.dll 0x7ffbff480000 0x7000 C:Program FilesMicrosoft SQL ServerMSSQL14.MSSQLSERVERMSSQLBinnSQLOS.dll Microsoft Corporation 2017.0140.3238.01 ((SQLServer2017-CU14).190913-2228) 9/14/2019 6:45:34 AM
SQLSCM.dll 0x7ffbf7e30000 0x13000 C:Program FilesMicrosoft SQL ServerMSSQL14.MSSQLSERVERMSSQLBinnSQLSCM.dll Microsoft Corporation 2017.0140.3238.01 ((SQLServer2017-CU14).190913-2228) 9/14/2019 6:40:48 AM
sqlservr.exe 0x7ff7869e0000 0x7f000 C:Program FilesMicrosoft SQL ServerMSSQL14.MSSQLSERVERMSSQLBinnsqlservr.exe Microsoft Corporation 2017.0140.3238.01 ((SQLServer2017-CU14).190913-2228) 9/14/2019 9:35:24 AM
sqlTsEs.dll 0x7ffbf91c0000 0x92f000 C:Program FilesMicrosoft SQL ServerMSSQL14.MSSQLSERVERMSSQLBinnsqlTsEs.dll Microsoft Corporation 2017.0140.3238.01 ((SQLServer2017-CU14).190913-2228) 9/14/2019 9:35:33 AM
svl.dll 0x7ffbf8b50000 0x2e000 C:Program FilesMicrosoft SQL ServerMSSQL14.MSSQLSERVERMSSQLBinnsvl.dll Microsoft Corporation 2017.0140.3238.01 ((SQLServer2017-CU14).190913-2228) 9/14/2019 6:57:13 AM
xplog70.dll 0x7ffc2ea50000 0x14000 c:Program FilesMicrosoft SQL ServerMSSQL14.MSSQLSERVERMSSQLBinnxplog70.dll Microsoft Corporation 2017.0140.3238.01 ((SQLServer2017-CU14).190913-2228) 9/14/2019 6:55:14 AM
xpsqlbot.dll 0x7ffbf7cd0000 0x8000 C:Program FilesMicrosoft SQL ServerMSSQL14.MSSQLSERVERMSSQLBinnxpsqlbot.dll Microsoft Corporation 2017.0140.3238.01 ((SQLServer2017-CU14).190913-2228) 9/14/2019 6:38:05 AM
xpstar.dll 0x7ffbf7c60000 0x6e000 c:Program FilesMicrosoft SQL ServerMSSQL14.MSSQLSERVERMSSQLBinnxpstar.dll Microsoft Corporation 2017.0140.3238.01 ((SQLServer2017-CU14).190913-2228) 9/14/2019 6:47:53 AM
Non_working Server
<<<<<<<<<<<<<<<
HKRUNTIME.DLL 0x7fffea270000 0x2d0000 DeviceCSVVolume137MSSQL13.KCOM1S130EDAM01MSSQLBINNHKRUNTIME.DLL 1/1/1970 8:00:00 AM
OPENDS60.DLL 0x7ffffe550000 0x9000 DeviceCSVVolume137MSSQL13.KCOM1S130EDAM01MSSQLBINNOPENDS60.DLL 1/1/1970 8:00:00 AM
QDS.DLL 0x7fffeac80000 0xdc000 DeviceCSVVolume137MSSQL13.KCOM1S130EDAM01MSSQLBINNQDS.DLL 1/1/1970 8:00:00 AM
SQLEVN70.RLL 0x7fffe5180000 0x2f0000 DeviceCSVVolume137MSSQL13.KCOM1S130EDAM01MSSQLBINNRESOURCES1033SQLEVN70.RLL 1/1/1970 8:00:00 AM
SQLBOOT.DLL 0x7fffeb230000 0x2e000 DeviceCSVVolume137MSSQL13.KCOM1S130EDAM01MSSQLBINNSQLBOOT.DLL 1/1/1970 8:00:00 AM
SQLDK.DLL 0x7fffe97b0000 0x4c8000 DeviceCSVVolume137MSSQL13.KCOM1S130EDAM01MSSQLBINNSQLDK.DLL 1/1/1970 8:00:00 AM
SQLLANG.DLL 0x7fffddf90000 0x25c7000 DeviceCSVVolume137MSSQL13.KCOM1S130EDAM01MSSQLBINNSQLLANG.DLL 1/1/1970 8:00:00 AM
SQLMIN.DLL 0x7fffe0560000 0x24fa000 DeviceCSVVolume137MSSQL13.KCOM1S130EDAM01MSSQLBINNSQLMIN.DLL 1/1/1970 8:00:00 AM
SQLOS.DLL 0x7fffff390000 0x7000 DeviceCSVVolume137MSSQL13.KCOM1S130EDAM01MSSQLBINNSQLOS.DLL 1/1/1970 8:00:00 AM
SQLSERVR.EXE 0x7ff7c29d0000 0x68000 DeviceCSVVolume137MSSQL13.KCOM1S130EDAM01MSSQLBINNSQLSERVR.EXE 1/1/1970 8:00:00 AM
SQLTSES.DLL 0x7fffe5470000 0x88b000 DeviceCSVVolume137MSSQL13.KCOM1S130EDAM01MSSQLBINNSQLTSES.DLL 1/1/1970 8:00:00 AM
- We can see HKRUNTIME already loaded in our problematic SQL, so the next most possible DLL should be the HKEngine.dll HKCompile.dll.
- Then check them in your environment, they do have some problem in those DLLs, for example the size/the modified date.
- Then we rename those DLL, for example HKEngine.dll.Old, and then run repair SQL Server, and after that we confirmed 1 instance start successfully.
Solution:
========
HKEngine.dll HKCompile.dll has some problem when SQL Server trying to load them.
So rename them and then run SQL Repair, and new DLL will be generated, then SQL Server start successfully.
SQL Server fails to start and If you look at the SQL Server Error log you will find «Failed allocate pages: FAIL_PAGE_ALLOCATION» and SQL Server generating exception dump. Similar to the SQL Server error log below.
Note: This blog is applicable when you out get of memory error during startup (or) with event ID: 2019 in system event log. For general troubleshooting of SQL Server out of memory errors follow steps in Troubleshooting SQLServer Memory
{
2013-01-02 12:31:20.91 Server Microsoft SQL Server 2008 R2 (SP1) – 10.50.2500.0 (Intel X86)
Jun 17 2011 00:57:23
Copyright (c) Microsoft Corporation
Enterprise Edition on Windows NT 5.2 <X86> (Build 3790: Service Pack 2)
2013-01-02 12:31:20.91 Server (c) Microsoft Corporation.
2013-01-02 12:31:20.91 Server All rights reserved.
2013-01-02 12:31:20.91 Server Server process ID is 1583.
2013-01-02 12:31:20.91 Server Authentication mode is MIXED.
2013-01-02 12:31:20.91 Server Logging SQL Server messages in file ‘C:Microsoft SQL ServerMSSQL10_50.MSSQLWIKIServerMSSQLLogERRORLOG’.
2013-01-02 12:31:20.91 Server This instance of SQL Server last reported using a process ID of 9240 at 1/3/2013 7:31:20 PM (local) 1/4/2013 12:31:20 AM (UTC). This is an informational message only; no user action is required.
2013-01-02 12:31:20.91 Server Registry startup parameters:
-d C:Microsoft SQL ServerMSSQL10_50.MSSQLWIKIServerMSSQLDATAmaster.mdf
-e C:Microsoft SQL ServerMSSQL10_50.MSSQLWIKIServerMSSQLLogERRORLOG
-l C:Microsoft SQL ServerMSSQL10_50.MSSQLWIKIServerMSSQLDATAmastlog.ldf
2013-01-02 12:31:20.92 Server SQL Server is starting at normal priority base (=7). This is an informational message only. No user action is required.
2013-01-02 12:31:20.92 Server Detected 24 CPUs. This is an informational message; no user action is required.
2013-01-02 12:31:20.94 Server Address Windowing Extensions is enabled. This is an informational message only; no user action is required.
2013-01-02 12:31:27.33 Server Failed allocate pages: FAIL_PAGE_ALLOCATION 1
2013-01-02 12:31:27.33 Server
Memory Manager KB
—————————————- ———-
VM Reserved 1534584
VM Committed 51576
AWE Allocated 0
Reserved Memory 1024
Reserved Memory In Use 0
2013-01-02 12:31:27.33 Server Error: 17311, Severity: 16, State: 1. (Params:). The error is printed in terse mode because there was error during formatting. Tracing, ETW, notifications etc are skipped.
2013-01-02 12:31:27.33 Server Using ‘dbghelp.dll’ version ‘4.0.5’
2013-01-02 12:31:27.34 Server **Dump thread – spid = 0, EC = 0x00000000
2013-01-02 12:31:27.34 Server ***Stack Dump being sent to C:Microsoft SQL ServerMSSQL10_50.MSSQLWIKIServerMSSQLLOGSQLDump0008.txt
2013-01-02 12:31:27.34 Server * *******************************************************************************
2013-01-02 12:31:27.34 Server *
2013-01-02 12:31:27.34 Server * BEGIN STACK DUMP:
2013-01-02 12:31:27.34 Server * 01/03/13 19:31:27 spid 4344
2013-01-02 12:31:27.34 Server *
2013-01-02 12:31:27.34 Server * ex_handle_except encountered exception C0000005 – Server terminating
}
Why would SQL Server fail with out of memory error (FAIL_PAGE_ALLOCATION)during the startup? Only possible reason that I could think of is Paged or NonPaged pool is empty.
How to prove if my Paged / NonPaged pool is empty? Look at the system event log for the Event ID: 2019
You will find error in system event log similar to one you see below.
{
Event Type: Error
Event Source: Srv
Event Category: None
Event ID: 2019
Date: 2013-01-02
Time: 12:31:00 PM
User: N/A
Computer: MSSQLWIKIServer
Description:
The server was unable to allocate from the system nonpaged pool because the pool was empty.
}
Above error indicates nonpaged pool is empty, When Nonpaged pool is empty every application would fail. How to identify who is consuming Nonpaged pool?
Use poolmon.exe from windows support tools. (Steps are documented in This KB).
If you r OS is windows 2003 or above you can simple run the exe from command prompt and identify who is consuming (Leaking J) space in Paged / NonPaged pool.
Below is sample output of poolmon.exe which I collected from my test system
![clip_image002[4]](https://mssqlwiki.files.wordpress.com/2013/01/clip_image0024_thumb.jpg?w=670&h=968 "clip_image002[4]")
Memory consumption by each tag is printed in above output. After finding the tag which is leaking the memory (Highest bytes)identify the Driver which is using the tag by using find command or strings utility from sysinternals (search for TAG in drivers folder %Systemroot%System32Drivers). Once you identify the driver, check if there are any known issue with the driver or you may have to contact the vendor of the driver to identify why the driver is consuming large amount of pooled /Non-pooled memory.
If you liked this post, do like us on Facebook at https://www.facebook.com/mssqlwiki , join our Facebook group MSSQLWIKI and post your SQL Server questions to get answered by experts.
Thank you,
Karthick P.K |
My Facebook Page |My Site| Blog space| Twitter
Disclaimer
The views expressed on this website/blog are mine alone and do not reflect the views of my company. All postings on this blog are provided “AS IS” with no warranties, and confers no rights.
-
-
January 22 2018, 17:12
Жила-была мумия одного SQL Server 2014(12.0.5557.0). Бежала мимо уборщица, шваброй махнула сервер упал и как давай по-полу валяться.
Проблема
После аппаратного сбоя старует и падает служба MS SQL Server.
Диагностика
В журналах сервера странное: CHECKDB прошел без ошибок для всех баз, но служба стартовать не хочет.
«000-00-00 00:00:00.00 spid8s Recovery is complete. This is an informational message only. No user action is required.
2000-00-00 00:00:00.00 Server SQL Server is terminating because of a system shutdown. This is an informational message only. No user action is required.
2000-00-00 00:00:00.00 spid37s Service Broker manager has shut down.
2000-00-00 00:00:00.00 spid8s .NET Framework runtime has been stopped.
2000-00-00 00:00:00.00 spid8s SQL Server shutdown has been initiated
2000-00-00 00:00:00.00 spid8s SQL Trace was stopped due to server shutdown. Trace ID = ‘1’. This is an informational message only; no user action is required.
2000-00-00 00:00:00.00 spid8s Error: 17054, Severity: 16, State: 1.
2000-00-00 00:00:00.00 spid8s The current event was not reported to the Windows Events log. Operating system error = (null). You may need to clear the Windows Events log if it is full»
В жуналах ОС видим Event 17311, источник MSSQLSERVER:
«SQL Server is terminating because of fatal exception c0000005. This error may be caused by an unhandled Win32 or C++ exception, or by an access violation encountered during exception handling. Check the SQL error log for any related stack dumps or messages. This exception forces SQL Server to shutdown. To recover from this error, restart the server (unless SQLAgent is configured to auto restart).»
Решение
Микрософт ничего интересного не предлагает: проверьте базу (это на мертвом сервере) или обратитесь в поддержку.
На самом деле всё оказалось просто: вовремя сбоя была повреждена одна из баз и при попытке её смонтировать MSSQL впадал в истерику (вспоминаем CHECKDB). Пострадавшая база была вычисленна с помощью последовательного блокирования доступа к файлам баз данных (прекратить доступ учетки сервера к файлам БД или, банально, их переименовать).
PS Как всегда, поврежденна оказалась мелкая и уже никому не нужная базуля.
Recently, I had replied to a MSDN post regarding an Access Violation reported by a SQL Server instance. In this post, I shall elucidate on troubleshooting memory dumps generated by SQL Server exceptions and what can be done without the availability of private symbols for debugging.
I had written a post earlier on when a memory dump is generated in SQL Server. If a single Errorlog has multiple stack dumps from multiple exceptions, then the first thing to do is to investigate the first access violation reported in the series of dumps reported. It is also possible that an exception (eg:access violation) could be followed by another exception/assertion or a non-yielding condition which may be a side effect of the earlier access violation.
When SQL encounters an access violation, there will be SQLDumpXXXX.mdmp, SQLDumpXXXX.txt and SQLDumpXXXX.log files created in the LOG folder associated with the exception. The SQL Errorlog will report the occurrence of the exception as well.
- The .mdmp file is the memory dump which would be required by CSS to troubleshoot the issue.
- The .txt file is the symptom dump file which contains environment information along with other information depending on the nature of the exception.
- The .log file contains an extract from the SQL Errorlog when the exception was encountered. This helps in looking at what messages were reported in the Errorlog prior to the issue. Helps during post-mortem analysis if the current Errorlogs have already been recycled.
There can be two types of exceptions: one that is fatal enough to cause a SQL service termination or one that encounters the exception which terminates the thread which was encountered the issue.
Now what can be done in such a situation without having to look into memory dump file. The first step would be to check if the issue is a recurring behavior due to a particular operation to establish some pattern or if there is not identifiable pattern at all. If you look into the SQL Errorlog, you might be able to identify the input buffer which cause the exception (unless it’s an exception which terminated the SQL service, one such example is shown below).
Server * BEGIN STACK DUMP:
Server * spid 5728
Server * ex_handle_except encountered exception C0000005 – Server terminating
If the above snippet contains the T-SQL query which raised the exception, then you would want to execute the same query from a Management Studio query window and check if the exception re-occurs. If yes, then you have a valid repro scenario.
Example snippet from an Errorlog where the Access Violation didn’t cause a service termination:
* Exception Address = 0042B866
* Exception Code = c0000005 EXCEPTION_ACCESS_VIOLATION
* Access Violation occurred writing address 67192000
* Input Buffer 38 bytes –
* exec sp_updatestats
Next check if you can spot any external components loaded into SQL Server address space. I shall revert back to my MSDN post where I had spotted an anti-virus filter DLL loaded in SQL Server process address space.
Server * bcrypt 000007FEFC900000 000007FEFC921FFF 00022000
Server * PCTLsp64 000000000B1E0000 000000000B251FFF 00072000
Server * mswsock 000007FEFC840000 000007FEFC893FFF 00054000
Based on a few Bing searches, I was able to determine that the PCTLsp64 was part of a virus monitoring program from PC Tools. The next step would be to check if this external component can be moved out of SQL Server process address space. The fact that the DLL is loaded in SQL process address space doesn’t make the DLL a culprit. However, when troubleshooting such issues, it is important to eliminate all unknown or third party elements that could prevent us from narrowing down to the culprit. Other DLLs which can be commonly found in SQL Server process address space would be provider DLLs when linked servers are being used on the SQL Server instance. Additionally execution of Extended Stored Procedures (XSPs) could also load external DLLs into SQL Server process address space depending on the way the XSP DLL was coded. An easy way to identifying a XSP dll is searching for the word “using” in the SQL Server Errorlog. SQL Server reports the first instance of a XSP dll being loaded into SQL Server process address space.
Using ‘xpstar.dll’ version ‘2009.100.1600’ to execute extended stored procedure ‘xp_readerrorlog’
If you do find an external component loaded in SQL process address space, check the callstack reported in the Errorlog below the list of modules printed below to determine if the functions of the exception stack are associated with external component that you identified. For the issue reported in the MSDN post, I did find this:
Module(sqlservr+00000000017A954C)
Module(kernel32+0000000000099380)
Module(ntdll+0000000000096228)
Module(ntdll+0000000000014F48)
Module(ntdll+0000000000034F6D)
Module(ntdll+0000000000015B2C)
Module(ntdll+000000000004F638)
Module(ntdll+000000000004C8F4)
Module(ntdll+000000000004C822)
Module(PCTLsp64+00000000000040AD)
Module(PCTLsp64+0000000000004701)
Module(PCTLsp64+0000000000005434)
Now this makes my story a bit more compelling of removing the external component out of SQL Server process address space. Once that is done and if you have a repro condition which causes the issue, then it would be good to re-run your repro and check if the issue is actually resolved. Not all the exceptions are the same even though they are reported in the same manner.
If all the above hasn’t helped in narrowing down the issue, then check if you are on the latest available cumulative update for the version of SQL Server that you are using. If not, then it might be a good idea to update your instance to the latest available build to ensure that you are not hitting a known issue. If this is possible, then apply the latest cumulative update package and check if the issue re-occurs. I am cognizant of the fact that this might be an easy option due to change management restrictions on production environments. For such a dependency, my recommendation would be engage CSS to assist with a thorough investigation of the root cause of the issue.
As you can see from the steps outline above, you can look into a few things before you need to start poking into the exception memory dump using a Debugging tool.
Happy debugging!
Hi,
We have a VCS 5.1 for windows 2003 R2 Enterprise edition x64 with 3 nodes (one local cluster with 2 nodes and alternate site with one node for BRP)
Originally we created the vcshelper for cluster control and a domain user account for SQL Server service.
For these accounts the installer requested domain administrator privileges so we granted.
Now for security management reasons and allignment with HQ we need to reduce privileges on these 2 accounts so we were reffered to document:
http://www.symantec.com/business/support/index?page=content&id=TECH31331
We did this but whenever we reduce privileges on the accounts the sql server does not start. I present the sequence of events below.
Steps:
1.We removed administrator privilege from vcshelper and domain user account
2. We changed both accounts for Veritas and SQL server. We restarted service in one machine and worked fine
3.Then we failed over to the other node and the cluster was fine but SQL could not start sending error:
Failed allocate pages: FAIL_PAGE_ALLOCATION 1
Error: 17311, Severity: 16, State: 1. (Params:). The error is printed in terse mode because there was error during formatting. Tracing, ETW, notifications etc are skipped.
4.I retried removing the account and using Local system account but got same message
5. We put back privileges to VCShelper and domain user account
6. And with this I made a failover cycle between the two nodes and worked fine. (A to B and B to A)
PLEASE ADVISE ON WHAT TO DO TO ALLOW SQL SERVER TO START
PS: We haven tried the next text of failover to alternate site due we have this problem in local cluster
Thanks in advance