Dynamics F/SCM Customer Experience (CRM) Dynamics GP Dynamics Business Central (NAV) Dynamics SL Power Platform Fabric Copilot

[Scott Yergensen]

On Monday our AOS started to crash several times per day. This seems to happen primarily while running sales reports such as quote and sales  confirmation journals We engaged our partner and Microsoft support team. MS support has been quite slow in responding. When they finally did they suggested full compile, full CIL,etc. We complete that and then started testing reports with just 1-2 users in the system. We ran many reports for 20 minutes and then the AOS crashed again. 

I have this message in the error log. 

Object Server 01:
[Microsoft][SQL Server Native Client 11.0][SQL Server]Could not find stored procedure ‘MTI_DYNAX12R3_PROD_01_model.dbo.XU_IsModelIdInstalled’.
{ CALL [MTI_DYNAX12R3_PROD_01_model].[dbo].[XU_IsModelIdInstalled](?, ?) }
session 1 (-AOS-)

Any thoughts or suggestions? We submitted dump file and error logs to MS and are now playing the waiting game. 

We are on 2012 R3 which had been stable for a long time until Monday. Single AOS environment.

——————————
Scott Yergensen
MTI, Inc.
Salt Lake City UT
——————————

[Corey Vantilborg]

This seems obvious,  but what changed on Monday?   That error looks either network, or SQL permissions related.   The AOS is trying to access a stored procedue on the database and cannot.  

1. Does that stored procedue exist in your AX Model Database?
2. What do the permissions for that SP look like?  The AOS service account needs access.

The fact the AOS runs the report a number of times then fails,  makes me think the error is not strictly related to that report.  If you run no reports for 20 minutes,  then run one does it crash?

——————————
Corey Vantilborg
ERP Analyst
Tigercat International Inc.
Brantford ON
——————————
——————————————-

[Jeff Austin]

Be sure to check the timing of that message closely.  I see that one when the AOS starts and verified with a restart in one of our non-prod environments.

Do you see anything a little earlier in the log?  AOS crashes are pretty distinct in the event viewer.

Jeff

——————————
Jeff Austin
Application Specialist 5
Bush Brothers & Company
Knoxville TN
——————————
——————————————-

[Scott Yergensen]

Our AOS has been up for about 14 hours now. We are only about an hour into our business day. I expect it to crash still but am grateful for a bit of stability. 

Corey, Early on Monday our partner imported a “simple” customization that would check for duplicate customer PO requisition number and return an infolog telling user if it was the same. We tested in TEST environment without issue prior to deploying. The problem started two hours later. We instructed them to remove the code by noon on Monday but the problem persists.   

Jeff, I went back and checked the error log against my notes when we were testing. This is the message at the exact time when the AOS crashed. 

The Microsoft Dynamics AX Object Server 6.3$01-MTI_DYNAX12R3_PROD_01 service terminated unexpectedly. It has done this 4 time(s).

System

– Provider [ Name] Service Control Manager [ Guid] {555908d1-a6d7-4695-8e1e-26931d2012f4} [ EventSourceName] Service Control Manager

– EventID 7034 [ Qualifiers] 49152

Version 0

Level 2

Task 0

Opcode 0

Keywords 0x8080000000000000

– TimeCreated [ SystemTime] 2019-08-15T00:58:35.815233700Z

EventRecordID 188053

Correlation

– Execution [ ProcessID] 748 [ ThreadID] 6604

Channel System

Computer

Security

–

EventData

param1

Microsoft Dynamics AX Object Server 6.3$01-MTI_DYNAX12R3_PROD_01

param2

4

41004F005300360030002400300031000000

——————————
Scott Yergensen
MTI, Inc.
Salt Lake City UT
——————————
——————————————-

[Corey Vantilborg]

This is going to sound like I’m preaching a bit, and I apologize for that. 

A couple of things concern me your reply. 

1. It sounds like your partner used an XPO to import code into your production system.  If this is the case it is unsupported and unsafe.  The only safe way to get new code into AX2012 is via a ModelStore import and deploy process.  If a full modelstore import was done,  do you know the process?  Was a DB Sync completed before any users logged in?
2. Don’t trust your partner on this,  it is your environment.  
3. I don’t believe in coincidences, that imported code is probably the source of your issue. 

Do you have modelstore back-ups? Or backups of the Model database?   I would be restoring the model database to a known good state.  

Finally,  when the crash happens, it is certain users only? If that is the case clearing Usage Data and their .AUC cache files may resolve it.  You could also clear the AOS caches,  but the server restart should have done that already.  

——————————
Corey Vantilborg
ERP Analyst
Tigercat International Inc.
Brantford ON
——————————
——————————————-

[Scott Yergensen]

Corey, Thank you for the response. Even if you are preaching it is 100% justified in this case. I am quite disappointed in the partner and myself for allowing this to happen. AX was already deployed when I started here and customizations already in place. We have not deployed many new customizations in my time here. It is clear to me that better procedures must be in place prior to any future customizations. 

1. I am checking with the partner to determine the method used. I will post back here shortly. 
2. Hard life lesson learned. Too much trust was placed in the partner. This will not happen again.
3. No way in the world this was a coincidence. 

Microsoft engineer just asked for a meeting in 20 minutes so hopefully we will get somewhere on this shortly.

——————————
Scott Yergensen
MTI, Inc.
Salt Lake City UT
——————————
——————————————-

[Jeff Austin]

If you have Premier/Unified support I’d start escalating with your TAM.  If you don’t have support I’d start with your Account Exec.  We’ve had mixed results with support but I’ve found if you can get the issue raised within Microsoft they are pretty good to respond.

I think a crash dump analysis could be the place to start.  I’ve seen some results come back pretty quickly once you can get someone on it.

Get with your partner and determine the exact contents of their payload and verify that they in fact removed it in it’s entirety (nothing sneaky hanging out in the USR layer, etc), performed a complete CIL compile (I always clear the xppil directories on each AOS) and DB synch, and restarted every AOS.

I fully agree with Corey – we will only XPO into a non-prod environment – and – there are no such thing as coincidences.  Any update to PROD requires a downtime where we can perform a modelstore import and completely reboot the system.  A modelstore database restoration may well be in your future.

——————————
Jeff Austin
Application Specialist 5
Bush Brothers & Company
Knoxville TN
——————————
——————————————-

[Dave Phillips]

Hi – 

Fargo picked up the support case.  You’ll hear from one of the best crash experts soon – Larry.

Thanks….Dave

——————————
Dave Phillips
Sr Support Escalation Engineer
Microsoft
Fargo ND
——————————
——————————————-

[Corey Vantilborg]

Having been helped by Dave and Larry a number of times in the past,  you are in great hands.

——————————
Corey Vantilborg
ERP Analyst
Tigercat International Inc.
Brantford ON
——————————
——————————————-

[Scott Yergensen]

Thank you. Just spoke to Larry. We are following his recommendations. I will update this as we learn more.

——————————
Scott Yergensen
MTI, Inc.
Salt Lake City UT
——————————
——————————————-

[Scott Yergensen]

Larry reviewed the dump files. He believes we had an issue with corrupted syslastvalue records. We reset usage data for all users per his suggestions during a break period for production this afternoon. So far we are stable. I will stress test tonight. I am grateful for the responses here and for getting assigned to Larry. It is refreshing when you land on someone who really knows their product. If I meet him at Summit I’m buying him a beer or coffee. Same with you Corey, Jeff and Dave.  Thanks again.

——————————
Scott Yergensen
MTI, Inc.
Salt Lake City UT
——————————
——————————————-