Dynamics F/SCM Customer Experience (CRM) Dynamics GP Dynamics Business Central (NAV) Dynamics SL Power Platform Fabric Copilot

[Neil MacDonald]

Hello all, I am finding our D365 F&O environment runs VERY slowly. In some cases, it can take 30 seconds to open a screen, other times it can take up to 2 or 3 minutes! As you can imagine, it makes any work very time consuming. I am being told that that is just the way Azure works. (We access it via Microsoft Dynamics Lifecycle Services).  Is it possible that here are different levels of Azure? 

Thanks in advance!

——————————
Neil MacDonald
Solutions Consultant
Binary Stream
Middle Sackville NS
——————————

[Glenn Linaa]

Hi Neil.
Do you still encounter slow performance or have you found the cause?

You don’t mention if the problem is happening once every day or continuously during the day. I have experienced after we shutdown the virtual machine for the night, that every task, form etz. is slow because nothing is cached. Onces I have opened e.g. purchase orders, then it will open much faster next time you perfom the operation. 

What we have done so far is making a warmup script that runs through various forms on the startup. This helps a lot every morning. 🙂 

Best regards

——————————
Glenn Linaa
Tasklet Factory Mobile WMS
Aalborg, Denmark
——————————
——————————————-

[Leonardus Smits]

Dear all, We have spent quite some time on managing serious performance issues we encounter and encountered at our Polish site. A few take aways (there are a lot of details, hot fix numbers, specific code in WMS area, batch setup but I’ll keep it to the highlights):
– Sizing, i.e. the number of AOS assigned. We were supported last year December by the Msft FastTrack Team. When going through the process of the sizing estimator, we kept getting errors. So in the end the sizing was done manually. Turns out though that the sizing wasn’t done based on the information we provided through our implementation partner at the time but based on the wrong sizing sheet uploaded in LCS. Getting that changed, it’s a real, real pain. We are still working on that one. The good news though is that apparently since a couple of months it is not necessary to have a downtime anymore to add AOS to a productive environment. This is information we found out that is not known throughout Msft, e.g. the support team didn’t know about this until yesterday.
– Batch jobs. Sizing became an issue also because we have a fair amount of batch jobs running. One of the solutions suggested is to split the batch load and the user interaction over different AOS. We’re in the process of doing so (dependent on the availability of additional AOS).
– Hot fixes. Probably the most important. In LCS you will see how many (binary) hotfixes there are which you haven’t installed. I’m referring here specifically to the X++ critical updates. But next to those, for which we now were advised by Msft to install all, in our case there are three critical updates that address specifically our issue with the runtime on batches in warehouse inventory management. Here we would have never gotten to the level of detail and support if we didn’t have a very good support partner (HSO) that has the detailed knowledge that matches or surpasses the knowledge of the Msft people. It took a while but the issue is now taken very seriously by Msft and they now also working on a backport for one hotfix (from 7.3 to 7.2 – we are on 7.2 and in the process of upgrading to 8.0.3, but that’s another story). 
– Support. Contracts, processes. First thing our Msft account executive keeps telling me: the standard support included in your Dyn365 contract is worthless. You need at least premium support. We have premium support via our support partner. But that’s just a very small part of this part of the challenge. Using the right procedure in combination with the right contract-element is an art in its own. Unfortunately not all people on the Msft-side are fully or correctly trained. So we lost a lot of the wrong emotional energy and time on pushing a service request that in the end then was simple denied by Msft. Without some inside knowledge from our support partner we probably would have ended up in a Kafkian novel and I would most likely have decided to take our Dyn365 to on prem. Now we keep willing to give it a go.
In summary, it is clear that knowledge on the management of the cloud services within Msft is still scarce and processes are not fully there were they should be. You will have to – either yourself or via a partner – really build up knowledge on the support structure and implement a system landscape that is considered best practice. The latter means – in our case – way more environments than we originally calculated for our business case when deciding to move from SAP to Dyn365. All manageable but for us it has not been as simple as we had – maybe naively – expected.

——————————
Leonardus Smits
SIGVARIS
Winterthur
——————————
——————————————-

[Ganapathi Sadasivam]

D365 F&O is a full-fledged ERP solution with parameters and knobs at every place for the customer/partner to tune the application to their fine needs. Over time these knobs will be less necessary as we evolve the product but right now they are still there and need to be used. Also on top, it allows the partner and ISVs to extend functionalities /customize features. All of them play a role in performance affecting positively or negatively and in most case the latter. There were several issues touched in this thread. I will take a shot at giving my perspective.

1. 30 Seconds to open a screen. Sometimes 2 to 3 minutes. Is it the way azure works?

It is NOT common for forms to take 30+ seconds to open. If it consistently takes over 30 seconds then these forms definitely need tuning. The query that gathers the data for the form may be sub-optimal. May be the it gathers more data than its needed or doing large amount of aggregation or using a poorly written XDS policies. In this case, the customer/partner should take a trace and find out what is causing poor performance and address it, as it will be painful to work with forms that take so much time – every time.

If these forms are not consistently slow, then the problem could be different. Excessive restarts triggering cold runs, blocking with concurrent workloads (though blocking unlikely cause slow forms (generally read), but could play a role during updates), or affected by high DTU (queries consuming most of the resources), or queries picking up a suboptimal execution plan are some possible causes. These things would need to be researched via telemetry which would be exposed via LCS, the raw log and SQL Insights.

The next question is what role Azure plays here? Azure as a platform provides many necessary features like scale, support and redundancy that D365 F&O takes advantage of. D365 F&O does not cut corners and it uses the best possible solution available to avoid problems like noisy neighbor issues. But like any cloud solution, it is subjected to some interferences too like latency surge, planned maintenance and critical updates. So for example when SQL team applies a critical security batch, the SQL will failover creating a blip and D365 may have to restart some of its services and user may see a slight degradation in their performance before it returns back to normal. We are always working with Azure and the SQL team to optimize this experience.

2. Edge Vs Chrome.
There was a point of time when Chrome clearly outperformed Edge for D365 F&O client interactions. But this is no longer the case. Internally we started using Edge on more consistent basis. But if there is a scenario where you see Chrome clearly gives you better performance, you should use Chrome if your org policy supports chrome.

3. Power BI refresh
Some of the routine activities can play a big part in consuming excessive amount of SQL resources. Power BI, Cube processing, export of data etc fall in category. Most of the time the complexity of the entity plays a big part in each of them. First ensure the entity you are pulling the data performs well and also ensure, you refresh or process only when it is needed. This will allow the other process to make use of the necessary SQL resources and does not have to compete for resource against the routine frequent activities.

4. Caching and warm-up.
D365 F&O employs a variety of caches to speed up things. Metadata cache, statement cache, AOS data cache, SysGlobalObjectCache and SysDataCache are a few in the AOS side. Of course SQL would use its own Procedure cache and buffer cache to speed things up. In the event of a planned or unplanned restart to SQL or AOS, the cache gets invalidated and may take a bit of time to prime these caches. There are several improvements made in AOS tier to reduce the necessity to restart. But still the restarts can’t be totally eliminated.

5. Sizing
Sizing questionnaire is the best way we could estimate the hardware need in short of doing a full benchmark. The formulas were written to map the answers from the sizing questionnaire to known benchmark numbers to estimate the hardware. It is unfortunate the formula triggered some errors in some cases. Microsoft would look into it. Also Microsoft is making an effort to tie Machine Learning algorithms to make this estimate accurate. So in future this will get better and avoid the number of times where we have resize the hardware after going live.

Microsoft monitors telemetry regularly and infer issues and work on necessary mitigation and fixes. But if the performance affects you consistently from doing your business, please open a case and work with the support. Microsoft dev team is committed to provide the best support it can.

6. Support
When the support cases are addressed by some experienced engineers, you may find the whole experience relevant without any wasted effort and quick. Microsoft is trying various ways to improve the support experience from added automation, early detection, self-healing and training our engineers on ongoing basis.

——————————
Ganapathi Sadasivam
Microsoft
Redmond WA
——————————
——————————————-

[Zvika Rimalt]

Hi Ganapathi,

“2. Edge Vs Chrome.
There was a point of time when Chrome clearly outperformed Edge for D365 F&O client interactions. But this is no longer the case. Internally we started using Edge on more consistent basis. But if there is a scenario where you see Chrome clearly gives you better performance, you should use Chrome if your org policy supports chrome.”

Our slowdowns with Edge are severe, all encompassing (on each and every form) and very very current. by no means a matter of the past.
To my non-technical eyes it looks there is some scripts/event storm in the background, where so many of them try to run, even data entry slows down to a halt?

We are not on the very latest build of D365 though – do you think this has only been resolved in the spring release?

——————————
Zvika Rimalt
Functional Consultant
Vancouver BC
——————————
——————————————-

[Mandy Kartes]

In regards to the commentary about Support, we are experiencing higher than forecasted customer adoption which has created some challenges in staffing and ensuring we have the right expertise available when our customers & partners need it.  We understand this puts our customers and partners in a difficult position. We have been actively growing our support team to overcome these challenges while also working to improve the support experience via process improvements, added automation, early detection, and self-healing/self-support. 

 

——————————
Mandy Kartes
Microsoft
Support Manager
Fargo ND
——————————
——————————————-

[Sandra Rudloff]

?We went live in April, and we are experience the very same.  It is sporadic, and we haven’t found the smoking gun.   We can type something and each character…… appears….. a …. few…..seconds……later. 

One thing to look at is your batch jobs – you might want to schedule all during non-business hours, if at all possible. 

Looking forward to hearing more feedback, and solving all our slowness issues!

——————————
Sandra Rudloff
Vice President
Pivot Interiors, Inc.
Santa Clara CA
——————————
——————————————-

[Zvika Rimalt]

we experienced the same awful slowdown when trying in  D365? and each letter takes few seconds to show up.

We discovered that ditching Microsoft Edge browser and using instead Chrome solves that particular problem.

Ironic, I know, but such is life.

——————————
Zvika Rimalt
BSA Finance
Vancouver BC
——————————
——————————————-

[André Arnaud de Calavon]

, ,

Our customer had the same issues using the Edge browser. Using Chrome or Internet explorer solved the slowness in this area also for us. I do wonder what is the culprit with the Edge browser.???

——————————
kind regards,

André Arnaud de Calavon
Solution Architect, Microsoft MVP – Microsoft Dynamics Business Solutions
——————————
——————————————-

[Francis Schnyder]

As  described above, we faced severe performance issues, and defined together with our support partner and Microsoft a list of actions to be taken (install newest hotfixes, re-schedule batch jobs?, install an additional AOS). 

However, what really made the difference (performance is very good now) is that Microsoft put our Azure SQL database to a higher service tier. Unfortunately, they refuse to tell us on which tier we are now and which it was before. This is not very customer friendly. However, we can attest that the support from Microsoft in this case was very good (once we reached the right contacts).

SQL performance can be monitored in LCS. Before the upgrade to the higher tier, SQL utilization was always over 50%, sometimes up to over 90% (in those moments it was not possible anymore for the users to work). Now SQL utilization is under 10% with peaks still under 20%.

——————————
Francis Schnyder
D365FO Global Application Manager SCM
SIGVARIS Group
Winterthur, Switzerland
——————————
——————————————-

[James Hnasko]

We experience a similar issue on-premise.  We ended up removing a AX server from the cluster and pinning the batch jobs and SSRS reports to render of that server. We also deployed a secondary SQL server and deployed the SSRS on the secondary. This way the inputs/outputs are segregated.  The batch jobs also need to be schedule and/or stagger  in times where application usage is low.  Storage latency was also a concerned. We migrated AX to SSD drives. You may not be able to solve the issue the same way as we did. However, from my experience, AX application slow downs and SQL locking usually occur from simultaneous reports / batch jobs running.

——————————
James Hnasko
Network Operations Manager
Hamamatsu Corporation
Bridgewater NJ
——————————
——————————————-

[Scott Morley]

Neil,
The production environment for D365FO runs on the same hardware for everyone. Microsoft monitors the Azure performance and adjusts instance configurations accordingly. If you are routinely seeing slowness then you can create a case with them to review it. There are a few things you can check yourself, as what you describe sounds more like a connection issue and not an Azure back end issue.
In LCS you can check some of the performance by looking at the diagnostics that are loaded there. You can see number of connections, SQL blocking/locking etc. To see if there is a connectivity issue (slow bandwidth, etc.), you can follow through the post below. As with any performance issue, it is best to start tracking when things are slow. Is it a certain time of day? Only specific users or functions? Having this in hand will help in any communication with Microsoft about the issue. 

Verify network capacity and throughput for Dynamics 365 Customer Engagement clients

Microsoft		remove preview
Verify network capacity and throughput for Dynamics 365 Customer Engagement clients In this article Applies to Dynamics 365 (online), version 9.xApplies to Dynamics 365 (online), version 8.x The primary characteristics of a network that affect the performance of Dynamics 365 clients, such as the web application or Dynamics 365 for Outlook, are bandwidth and latency. Bandwidth is the width or capacity of a specific communications channel. View this on Microsoft >

——————————
Scott Morley
Principal Application Architect
OneNeck IT Solutions
Bend OR
——————————
——————————————-

[Ludwig Reinhard]

Hi,
Have you already contacted the MS support and reported that issue?
Do you experience the slow performance in your live or test systems?
Best regards,
Ludwig

——————————
Ludwig Reinhard
Sycor
Goettingen
——————————
——————————————-

[Paul Martin]

?Currently we have a case open with Microsoft, for us the system performs well b ut we have found that after some period of time and it is not predictable, if you are using the Edge browser performance can be a real dog. While working on this with Microsoft, we have been having our users use the Chrome browser which then performs like Edge used to before it acted up. Not sure this helps as for us it seems to be something with the browser, but anyway you might give Chrome a try to see if it helps in your situation.

——————————
Paul Martin
Production Program Manager
Elite Comfort Solutions, LLC
Conover NC
——————————
——————————————-

[Jo Olson]

we have found that PowerBI refresh schedule was causing a serious slowdown. I’m not sure if this applies to you but though I would throw it out there that we are creating a data warehouse to relieve some of this problem along with changing the refresh schedule to less often.

——————————
Kimberly Jo Olson
Internal Project Manager
Pavement Recycling Systems, Inc.
Jurupa Valley CA
——————————
——————————————-