Gatsby RSS Feed

X++ Show Cross-Reference With Read/Write field reference

Tue, 24 Mar 2026 22:12:03 GMT

From the first release of Visual Studio for X++ development, one of the top ideas was to improve cross-references - specifically, to add a Read/Write modifier in the Cross-Reference form:

I don't know why Microsoft has not been fixing this for so long. In the latest release 10.0.47 they introduced a new Cross-reference form (with Model and Type grouping), but still don't address the main issue: Read vs Write references for table fields.

This is quite a typical task in X++ development: you have a table field and want to know which code updates this field.

Recently, I managed to connect Claude Code to the TRUDUtilsD365 DevTools project, and it was able to implement this feature(The main challenge was teaching Claude how to work with standard D365 DLLs - once that was sorted, it handled the rest).

So a new release of TRUDUtilsD365 DevTools includes a "Show Cross-Reference" Add-in that displays Read/Write access type for field references.

The idea is simple: find all code references to a field, extract the code line, and analyse whether it's a Read or Write. Currently, the detection looks for the "=" symbol - if the field is on the left side, it's a Write; everything else is Read. It also handles patterns like setField(fieldNum(...)).

I tested it on several popular fields, such as SalesLine.SalesPrice and it works great. The tool also displays the actual code line (configurable - one or several lines of context), so you can visually confirm what the code does.

View for 1 line:

View for 3 lines:

Other features:

Supports tables, classes, forms, views, data entities, enums, methods, and extensions
Filter by module and access type
Go to source navigation (opens the X++ file at the exact line)
Copy to clipboard (paste-friendly for Excel)

The tool is open-source and 100% AI-written, so if you're missing something, you can ask Claude to add it.

I hope you find this tool useful. Install it from here

Feel free to ask questions! (The original post for this tool is here: https://www.linkedin.com/pulse/x-show-cross-reference-readwrite-field-reference-denis-trunin-0p8ec ).

D365FO License usage log utility

Sun, 08 Mar 2026 22:12:03 GMT

In this blog post, I will describe how the current licensing works in D365FO and introduce an open-source X++ utility that allows you to track the actual usage of elements and compare it with the elements assigned to users via user roles.

Understanding the current licensing model

At first, the new licensing model may seem confusing. For example, what does "Entitled" or "not Entitled" mean? Here, I will describe my journey to understanding it.

The legacy model relied on AOT properties for MenuItems. For example, opening the WHSLoadTable MenuItem in Visual Studio shows two properties: one for the read-only license, and another for the write license.

This model obviously couldn't cover all the complexities of the current license model (e.g., whether you have Finance, SCM, or both). As a result, Microsoft released a new model where they store all the license information on their servers and provide it to users in Licensing* tables. The data for these tables is calculated internally by Microsoft services and periodically (e.g., every 8 hours) synchronized with the Tier 2 or PROD environments. In this model, all custom menu items are treated as Teams licenses.

Here are the license properties for the same menu item under the new model.

The logic is now different: if a user has Read access to this WHSLoadTable menu, they must have one of the 7 specified licenses. If they have Write access, they must have an SCM or SCM Premium license. This is exactly what "Entitled" means.

Microsoft provides a form "Licenses usage summary" which displays the results of the above query, but in a different level of granularity (Role - Duty - Privilege)

Special roles

There are special roles that are exempt from the license report:

System administrator - This role has access to everything and is currently excluded from the license report.
Device based licenses - This is a common scenario in warehouse environments. For example, if you have a physical computer on the warehouse floor and five warehouse workers who periodically use it to perform various activities, instead of licensing these five people individually, you license just one device. You then assign the "Device-based" role to these users, which excludes them from the license report. It's my understanding that this is not currently strictly controlled by Microsoft, but they plan to introduce some level of control in the future.

After familiarizing yourself with the "Licenses usage summary" form, I also recommend reading this series of articles by André Arnaud de Calavon, who has done a fantastic job of explaining various security topics. This will give you a solid foundation in the new licensing model.

There are several companies that focus on D365FO security and licensing optimization. When I originally asked a question about user logging on LinkedIn, I received a lot of insightful comments. I've saved them here for reference:

André Arnaud de Calavon / Next365 — D365 expert and consultant, probably the top writer regarding security topics. Link: André Arnaud de Calavon
D365RoleSecure / NoirSoft — D365FO security and licensing-focused tool. Link: NoirSoft / D365RoleSecure
Fastpath Assure / Alex Meyer — Security, audit, compliance, and D365FO licensing governance. Link: Fastpath Assure
Marco Romano / IT|Fandango — D365 F&O consultant offering licensing/right-sizing assessment services. Link: IT|Fandango
XPLUS Process Discovery / Anthonio Dixon — User activity and process discovery for D365, with license optimisation angle. Link: XPLUS Process Discovery

License usage log tool

The main limitation with Microsoft's standard solution is that it doesn't provide an actual activity-based license usage report.

Consider a simple scenario: 10 users share a single role. This role contains 100 menu items that only require a Teams license, and one menu item that requires an Activity license. How can we determine which of these 10 users are actually using that specific Activity menu item?

The License usage log tool was built to fill this gap and answer exactly these types of questions. It can be downloaded from GitHub and deployed to your PROD environment using your standard X++ pipeline.

Enabling element usage logs

After installation, you need to enable element usage logging. You have two options:

Full (Debug only) mode - Every single access is logged.
Summary mode - The system creates an element usage log entry for a user only once per session.

You should run this for a couple of weeks in your PROD environment to collect meaningful usage data.

The following events are logged:

Form openings (using the same extension point as the standard Microsoft "Form runs (Page views)" telemetry)
SysOperation executions (including reports and actions)
FormLetter executions (such as sales and purchase order postings)
RunBase class executions

Calculating data modifications

One of the main challenges of license monitoring is distinguishing whether a user only viewed a form or if they actually modified data. We can gather modification events from two sources:

Table ModifiedBy and CreatedBy fields - This is not always fully reliable, as it only records the last user who touched the record.
SysDatabaseLog table - This provides more accurate event information, but it must be enabled in advance.

The tool allows you to specify a period (e.g., the last 90 days) and process the modification information from the two sources mentioned above.

The next challenge is linking the form to a list of tables. The License tool automatically calculates this by linking all form DataSources with the corresponding MenuItem, but you can also manually correct these links if needed.

Update 20.04.2026: There is also a v2 of this tool that supports update logging from the form (https://www.linkedin.com/pulse/d365fo-license-log-utility-v2-improved-data-logging-denis-trunin-um6fe/)

You can enable write operations logging:

When enabled, the system dynamically tracks user modifications using event handlers on form data sources:

Event handlers are attached to form Insert, Update, and Delete events
A log entry is created whenever a modification occurs
Only one record per user session per table is logged
Logging can be enabled for specific users only

Service functions

The License log form includes a couple of useful service functions:

Cleanup function: Deletes all log records older than a specified period (e.g., 90 days) and compacts the remaining data, leaving only one record per user, per element, for the entire period.

Comments per user: The reporting table is recalculated on every run, but you can add free-text comments that are preserved between sessions (for example, to note that a specific user has already been validated).

It also includes a couple of custom recalculated tables, which are primarily used for tracking and debugging:

RunBase items: Links MenuItem names with their corresponding RunBase class instances. Since RunBase logging only captures the class name, this table is required to link the class back to its originating MenuItem.
Permissions: A view containing MenuItems with their Read and Write license requirements.

Running the license usage report

After collecting the element usage log and calculating the data modification information, open the User license report tab and click Refresh data. The report will compare each user’s currently assigned license with the highest license tier required based on their captured system usage.

The report consists of two sections: a header and lines.

Header section

The header contains one row per user and provides their licensing status:

User – The user account being analyzed.
Enabled – Whether the user account is currently active.
Last logon date – The latest detected sign-in date for the user.
License – The license currently assigned to the user.
Usage log license – The highest license level required based on the Element Usage log.
Compare Status – A comparison between the assigned license and the usage-based license.
High-tier lines – The number of records that contribute to the highest detected license requirement for that user.
Comment – An optional, custom reviewer comment.
High-tier objects – The specific objects that drive the user into their highest detected license tier.

Lines section

The lines section details the individual securable objects behind a selected user's result, explaining exactly why they fall into a specific license tier:

AOT Name, Type – The AOT name and type of the MenuItem.
Read License – The license tier necessary for read access.
Write Permission – The license tier necessary for write access.
Access – The access level currently being evaluated for the user (Read or Write).
Current Permission – The permission currently assigned to the user via security roles.
Writes – Indicates whether actual write activity was verified for this object based on the collected data modification logic.

Analyzing the license report data

This report evaluates users by comparing their Assigned license level (the "License" column) with their Actual system usage (the "Usage log license" column), based on captured daily activity and entitlement objects.

Let's review the possible analysis scenarios based on the Compare status field:

1. Match

The assigned license accurately corresponds to the user’s actual system activity.

Technical meaning:

The user is accessing allocated menu items.
Logged operations confirm the required access level (including write access where applicable).
The assigned SKU properly aligns with the required entitlement objects.

Example 1

In this example, we can see that the user accesses the Work and Waves forms and actually writes to the relevant tables. This legitimately requires a Supply Chain Management license; it cannot be further optimized.

Example 2

This example is a bit different. The user only runs the TMSPACKINGLIST report. Microsoft recently changed this report to require an Enterprise-level license (it previously only required an Activity license). Since the high license requirement is caused by this single report, it presents an opportunity to reduce costs by developing a custom report or providing the required data through a different channel.

A common question here is: Can we just duplicate the standard MenuItem and use our custom copy instead? According to the D365 licensing guide, this practice constitutes "Multiplexing," and still requires the original underlying license to be applied.

2. Match – Write Not Confirmed

The user accesses functionality corresponding to their assigned license, but no write operations were captured for the related tables.

Technical meaning:

The user accesses the menu items.
Their assigned access level matches their allocated license.
However, no actual database writes were logged for those sessions.

Example

The user accesses the Waves form, but there is no confirmed record of them writing to its underlying tables. While this requires further investigation, it highlights an opportunity to lower their access to this form to read-only, which could potentially reduce their required license level down to a Teams license.

3. Higher Than Required

The user’s assigned license is higher than what their actual logged system activity.

Technical meaning:

Logged operations demonstrate a lower utilization level than the allocated SKU.
The required entitlement objects actually fall into a lower license tier.

Example

In this case, the user is assigned a full SCM license. However, the usage log reveals they only use two forms that require an Activity license. This makes them a strong candidate for a permissions review to right-size their licensing.

4. Higher Than Required (No Activity)

The user has an assigned license, but zero recorded system activity.

Technical meaning:

No logins or operational activities were captured in the system logs.
No entitlement usage was detected.

The user is assigned an SCM license, but no logging activity was recorded. There are a few possibilities here: the user may no longer require system access and should be disabled; or, the user might be a high-level manager who only needs periodic read access, or could perhaps be assigned a different appropriate role instead of holding an unused premium license.

Quick overview with AI

For a fast executive overview, you can copy the report's header data into Excel, then pass it to ChatGPT alongside this prompt to generate a helpful written summary for stakeholders.

Summary

A license usage log utility provides insights into how users interact with the system by producing a usage report that compares the required license usage based on actual activity against the currently allocated licenses based on user roles.

The tool is open source and can be downloaded from GitHub.

I look forward to your feedback on it. Specifically:

Do the currently logged operations provide a clear enough view of license usage, or is more data needed?
Do you have any guidance or best practices you can share with the community on how to adjust security roles based on this tool's output?

For general questions, please use the original post on LinkedIn. For bug reports or feature requests, please use the GitHub issue tracker.

I hope you found this post helpful. As always, if you have any suggestions for improvements or questions regarding this implementation, please don't hesitate to reach out.

Implement Service-based integration in D365FO

Mon, 02 Mar 2026 22:12:03 GMT

The External Integration framework is an open-source solution designed for inbound and outbound integrations in D365FO.

In this blog post, I will describe how to implement a service endpoint using the External Integration framework.

Key Design Principles

X++ services are created using a standard service class and the Service and Service group objects.

Please note that a service is a synchronous integration pattern that increases system coupling. It should only be used when an asynchronous approach is not an option.

Before starting development, I analyzed several projects to see how different developers implemented their services. My key observations were:

The input parameters vary by integration and cannot be universal.
The output parameters look very similar: typically a dataset containing one or more tables, along with some error flags. Additionally, most implementations included some kind of logging table, which usually varied per implementation.

Based on these observations, the following concept was implemented:

The service input contract will be a new class for each integration, providing maximum flexibility.
The output contract will be a unified class containing the following data: a .NET dataset (with 0 to n tables), an output string, an error flag, and an error message.
A common logging option will be provided.
It must be possible to test the class directly inside D365FO.

Implementation details: X++ code

I have provided sample code in the following class.

To create a new service using the External Integration framework:

Create a new class that extends DEVIntegServiceExportBase.
Create an entry point that is always a single line—just a call referencing the main method.
Implement the main method to contain only the service's business logic.

A code sample looks like this:

public class DEVIntegTutorialServiceTest extends DEVIntegServiceExportBase
{
/// DEMO #1 (DataTables / DataSet):
	public DEVIntegServiceExportResponseContract getSOInfoDataTable(DEVIntegTutorialServiceTestContract _contract)
    {
        return this.serviceCallProcess(methodStr(DEVIntegTutorialServiceTest, calculateGetSOInfoDataTable), _contract);
    }

    public void calculateGetSOInfoDataTable(DEVIntegTutorialServiceTestContract _contract, DEVIntegServiceExportResponseContract _response)
    {
        this.validateAndDelay(_contract);

        changecompany(_contract.parmCompanyId())
        {
            SalesTable salesTable = SalesTable::find(_contract.parmSalesId(), true);
            if (!salesTable.RecId)
            {
                throw error(strFmt("Sales order %1 not found in company %2", _contract.parmSalesId(), _contract.parmCompanyId()));
            }

            this.initDataTableHelper();

            dataTableHelper.addDataTable('SOHeader');
            dataTableHelper.addRowItem('SalesId',     salesTable.SalesId);
            dataTableHelper.addRowItem('CustAccount', salesTable.CustAccount);
            dataTableHelper.addRow();

            // Lines table: multiple rows with sales lines
            dataTableHelper.addDataTable('SOLine');

            SalesLine salesLine;
            while select salesLine
                where salesLine.SalesId == salesTable.SalesId
            {
                dataTableHelper.addRowItem('SalesId',  salesLine.SalesId);
                dataTableHelper.addRowItem('ItemId',   salesLine.ItemId);
                dataTableHelper.addRowItem('SalesQty', salesLine.SalesQty);
                dataTableHelper.addRow();
            }

            this.addDataTableHelperToResponse(_response);
        }
    }

    /// DEMO #2 (Temp table / DataSet):
    public DEVIntegServiceExportResponseContract getSOInfoTempTable(DEVIntegTutorialServiceTestContract _contract)
    {
        return this.serviceCallProcess(methodStr(DEVIntegTutorialServiceTest, calculateGetSOInfoTempTable), _contract);
    }

    public void calculateGetSOInfoTempTable(DEVIntegTutorialServiceTestContract _contract, DEVIntegServiceExportResponseContract _response)
    {
        this.validateAndDelay(_contract);

        changecompany(_contract.parmCompanyId())
        {
            SalesTable salesTable = SalesTable::find(_contract.parmSalesId(), true);
            if (!salesTable.RecId)
            {
                throw error(strFmt("Sales order %1 not found in company %2", _contract.parmSalesId(), _contract.parmCompanyId()));
            }

            SalesTable tmpSalesTable;
            tmpSalesTable.setTmp();
            tmpSalesTable.SalesId     = salesTable.SalesId;
            tmpSalesTable.CustAccount = salesTable.CustAccount;
            tmpSalesTable.doinsert();

            this.initDataTableHelper();

            // Return temp table as a dataset DataTable (SOHeaderTmp)
            dataTableHelper.addTmpTable('SOHeaderTmp', tmpSalesTable,
                [
                    fieldNum(SalesTable, SalesId),
                    fieldNum(SalesTable, CustAccount)
                ]);

            this.addDataTableHelperToResponse(_response);
        }
    }

In case of errors, simply throw an exception, and the framework will handle it automatically. There are also a few helper methods available that can return either a temporary table or a dataset.

On the caller side, if the calling system is .NET-based, it needs to analyze the result by following these steps:

Check the IsSuccess flag, and review the error details in ErrorMessage if it failed.
If successful, deserialize the MessageDataSet parameter into a System.Data.DataSet object and read the required data.

Implementation details: User configuration

After creating a service class, it must be added to the External Integration > Setup > Service message type form.

From this form, we can also test the service class using the Test button.

The system will automatically create an empty contract. We just need to specify the parameter values and press the Execute button.

If the call is unsuccessful, an error message will be generated.

Logging

The framework provides different logging options tailored to various business use cases.

The following logging options are available:

None – No logging.
Statistics & Errors – Creates one summary record per day for successful calls (showing the total number of calls and lines processed), PLUS detailed records for all calls that encounter errors during execution.
Errors only – Only failed calls are logged.
Request – Logs each request, including its parameters, status, and statistics (start/end date and number of lines).
Full – Same as Request, but also logs the response payload (note: this may consume significant storage space).

Recommended values for production environments are "Request" or "Statistics & Errors".

Notes on consuming services

Services can be consumed directly by connecting to the D365FO instance using an AppId. However, in recent projects, we have utilized the Azure API Management service. Adrià has written an excellent article detailing this approach.

In short, you will need to create an HTTP endpoint:

https://testsystemlink.sandbox.operations.dynamics.com/api/services/

With a new operation:
Display name: getSOInfoDataTable
Name: getsoinfodatatable
URL:  /DEVIntegTutorialServiceTestGroup/DEVIntegTutorialServiceTest/getsoinfodatatable

and apply the following policy

<policies>
    <inbound>
        <send-request mode="new" response-variable-name="bearerToken" timeout="20" ignore-error="false">
            <set-url>https://login.microsoftonline.com/{{TenantId}}/oauth2/token</set-url>
            <set-method>POST</set-method>
            <set-header name="Content-Type" exists-action="override">
                <value>application/x-www-form-urlencoded</value>
            </set-header>
            <set-body>@{return "client_ID=4989d73b-1e25&client_secret=jSv8Q~m~jmO0QtZFmi_h4di&resource=https://testsystemlink.sandbox.operations.dynamics.com/&grant_type=client_credentials";}</set-body>
        </send-request>
        <set-backend-service base-url="https://testsystemlink.sandbox.operations.dynamics.com/api/services/DEVIntegTutorialServiceTestGroup" />
        <set-header name="Authorization" exists-action="override">
            <value>@("Bearer " + (String)((IResponse)context.Variables["bearerToken"]).Body.As<JObject>()["access_token"])</value>
        </set-header>
        <base />
    </inbound>
    <backend>
        <base />
    </backend>
    <outbound>
        <base />
    </outbound>
    <on-error>
        <base />
    </on-error>
</policies>

Next, test the API using the following parameters:

{"_contract": {"CompanyId":"USMF","DelaySeconds":1,"SalesId":"001036"}}

Alternatively, if you are using Postman:

https://d365apiserv.azure-api.net/mytest/DEVIntegTutorialServiceTest/getSOInfoDataTable

POST
subscription-key 
991874b1c..

Body raw
{
    "_contract": {"CompanyId":"USMF","DelaySeconds":1,"SalesId":"001036"}
}

Summary

The External Integration framework offers the following benefits for service development:

A standard output contract that includes workload statistics and error flags.
Automatic exception handling.
Flexible logging options.

All the classes discussed can be found on GitHub and serve as a great starting point for your own integrations.

I hope you found this post helpful. As always, if you have any suggestions for improvements or questions regarding this implementation, please don't hesitate to reach out.

Sync data from Dynamics 365 Finance & Operations Azure SQL Database (Tier2) to local SQL Server (AxDB)

Thu, 15 Jan 2026 12:00:00 GMT

One of the frequent challenges in Dynamics 365 Finance & Operations development is keeping the local development database (AxDB) synchronized with fresh data from a Tier2 (UAT/Sandbox) environment. Standard approaches often involve restoring a full BACPAC, which is time-consuming and overwrites everything, or manually copying data, which is tedious and error-prone. To address this, I created the D365FO-DBSync utility. This tool helps developers synchronize data from D365FO cloud environments to their local development databases, making it easier to test with production-like data.

The main idea is to make the last X records (ordered by RecId) the same between Tier2 and AxDB. E.g. A typical dynamics implementation has ~2000 tables with the data, around 200 tables are transactional tables that exceed 100k records, so if we sync last 100k records per table we can get a compact DB with the most of the base data and recent transactional data.

Usage

For detailed technical instructions and setup steps, please refer to the GitHub repository.

The main workflow is straightforward:

Prepare your environment: Whitelist your IP and get database credentials from LCS.
Configure the tool: Enter the connection details for both the source (Tier2) and destination (Local AxDB).
Define strategies: Choose which tables to copy and how (e.g., last 10k records).
Sync: Run the discovery and process steps to synchronize the data.

Custom Copy Strategies

The default behavior of the tool is to copy the last X records (based on RecId). While this works for most transaction tables, you might need more control for some tables.

You can define custom copy strategies using the following syntax:

TableName|RecordCount|sql:CustomQuery -truncate

also the CustomQuery may contains the following placeholders:

@recordCount: Replaced with the record count defined in the global parameters.
@sysRowVersionFilter: Used for enabling INCREMENTAL mode in custom SQL strategies.

Examples

Let's describe different scenarious where a custom copy strategy is required.

Copy tables where a row contains large data

Some tables may contains images or some binary info and copying a standard 100k records may take a lot of time. For such scenario one you can specify custom RecordCount to copy to limit the number of records. Usage examples:

*   `ECORESPRODUCTIMAGE|1000`: Copies the last 1,000 product images.
*   `WHSCONTAINERTABLE|50000`: Copies the last 50,000 warehouse containers.

Copy Inventory dimensions if Licence plate is used(InventDim)

In case you use a WHS solution, InventDim table may contains a lot of records with a LicensePlateId. During the copy you can't simply take the latest records, as you miss a lot of references to InventDimId that have a blank LicensePlateId. The solutoin for this is to take all dimensions without LicensePlateId PLUS all recent records with LicensePlateId. In this case you transfer the records that may be used in some setup and the latest transactional records. A setting for this looks the following:

    InventDim|sql: SELECT * FROM InventDim WHERE RecId IN (SELECT RecId FROM (SELECT RecId FROM InventDim WHERE LICENSEPLATEID = '' AND PARTITION = 5637144576 AND DATAAREAID = 'USMF' AND WMSLOCATIONID = '' UNION SELECT RecId FROM (SELECT TOP 50000 RecId FROM InventDim ORDER BY RecId DESC) t) u) AND @sysRowVersionFilter ORDER BY RecId DESC

Copy inventory onhand (InventSum)

If InventSum is large, you may deside to copy only records that have some values. These records has the foolowing filter (Closed = 0), and it can reduce data volume. A settings for this will be the following:

    InventSum|sql: SELECT * FROM InventSum WHERE Closed = 0 AND @sysRowVersionFilter ORDER BY RecId DESC

Copy inventory reservations

To optimize a copy of inventory reservation (WHSINVENTRESERVE) we can use a filter that includes all reservation for a level 1 PLUS any records modified in the last 93 days. This ensures you have both current availability and recent history.

    WHSINVENTRESERVE|sql: SELECT * FROM WHSINVENTRESERVE WHERE ((HIERARCHYLEVEL = 1 AND AVAILPHYSICAL <> 0) OR MODIFIEDDATETIME > DATEADD(DAY, -93, GETUTCDATE())) AND PARTITION = 5637144576 AND DATAAREAID = 'USMF' AND @sysRowVersionFilter ORDER BY RecId DESC

Copy Performance Optimization

Copy speed is optimized using multiple approaches.

SysRowVersion Optimization (Incremental Sync)

The tool saves a SysRowVersion for every table during the first run. If a saved value exists, it tries to estimate the changes first by querying only system fields before copying all data.

If changes are below a certain percentage(e.g. 40%), it deletes only changed records and re-inserts them.
If changes are higher, it truncates the table (to save time on delete) and re-inserts the table.

This may give a reduction in data transfer for tables with minimal changes.

Smart Strategies

You are not limited to just "all or nothing". You can exclude specific tables (e.g., Sys*, *Staging) or write custom SQL to fetch only relevant data (e.g., specific DataAreaId).

Execution Efficiency

Parallel Execution: Uses multiple workers to process tables concurrently. Usually 10-20 workers are used.
Bulk Insert: Uses SqlBulkCopy for high-throughput data insertion.

With all these changes, the time for the first run (e.g. limit 100k records) may be around 2 hours, and for subsequent runs it may start from 30+ minutes.

Future Usage Scenarios

With the recent announcement from Microsoft stating there will be "No New Cloud Implementation Projects in Dynamics Lifecycle Services" (LCS), the landscape for Dynamics 365 development is evolving. As the ecosystem moves towards the Power Platform Admin Center (PPAC), utilities like D365FO-DBSync may become helpful.

One of the limitations with the new PPAC experience is the inability to download a database backup directly(see Terminology differences between Lifecycle Services and the Power Platform admin center).

This limitation complicates the process for developers who rely on restoring UAT data to their local environments for debugging and development.

Furthermore, adoption of the Unified Developer Experience (UDE) remains low. A recent poll indicates that despite being introduced over two years ago, only about 13% of developers have switched to UDE.

As a result, it is likely that many developers will migrate to a hybrid workflow combining UDE for some tasks and local VHDs for others. In this mixed environment, D365FO-DBSync can assist with data transfer, helping developers work with relevant datasets.

How This Tool Was Built

It is an interesting time for software development. This tool contains around 3000 lines of C# code, but, I didn't write a single line manually. All of it was done using Claude Code, leveraging Claude 4.5 Sonnet for the actual coding and Claude 4.5 Opus for the planning phase.

In planning mode, I would provide requirements to Opus and ask it to generate questions to create a detailed specification. For a single feature, it might ask 50+ questions. Once the specification was ready, I would provide it to Claude Code for implementation.

The only downside is that the standard $20 Claude subscription is not enough; the session limits are reached quite quickly. You often need to switch to API mode, which can consume $5-10 per few requests, or wait for the limits to reset.

Summary

This tool helps reduce the friction of getting fresh data into your local VM.

You can download the D365FO-DBSync from the GitHub repository:
https://github.com/TrudAX/D365FO-DBSync

It is currently a working prototype, and it definitely requires input from different projects to become mature, so feel free to share your ideas on what can be improved.

D365FO Integration: Import Purchase Orders from PDF using Gemini AI

Wed, 10 Sep 2025 22:12:03 GMT

In this blog post, I'll guide you through the process of using AI to import purchase orders into Dynamics 365 Finance and Operations (D365FO) from complex PDF files.

Although the example provided here is simplified to illustrate core integration concepts, the approach and code samples are based on real-world scenarios. This makes them highly adaptable for similar integration tasks you might encounter. We'll utilize the free and open-source External Integration Framework, which provides reusable components explicitly designed for building robust integrations in X++.

Integration Scenario Overview

Let's start by defining the integration scenario:

Goal: The customer receives complex invoice documents from external vendors as PDF files. We want to create Purchase orders in D365FO based on these documents.

Current solutions on the market

Before approaching this task, I researched the available options.

PDF to Excel converters

There are numerous online services, such as https://studio.convertapi.com/pdf-to-excel and https://www.ilovepdf.com/pdf_to_excel, that claim to extract tables from invoice documents. The idea was that the user converts the PDF to an Excel template and then uploads it to D365FO.

However, after initial testing, they generated many errors in the line sections—for example, empty cells when a description spans two lines, or merged cells where columns should be separate, etc.

Invoice capture solution

The invoice capture solution is a standard option for Microsoft Dynamics 365 for Finance and Operations (D365FO). It is based on the Power Platform and utilises Optical Character Recognition (OCR) to parse invoices.

I researched it and found:

The installation document is 65 pages long, and you must be an administrator of “everything” just to try it.
Because it’s based on the Power Platform, extensibility is limited.
The Yammer group is full of questions (for example, why something is incorrectly parsed on page 2) without clear responses.

Because invoice capture relies on Azure services, I tried Microsoft AI Foundry Document Intelligence directly in Azure, and result's were not perfect even on the first sample (see below)

The advantage of OCR is that it provides a direct link between the image and the parsed data, and it is quite fast. However, after the initial research, I decided not to pursue this further.

Use AI models

We currently live in an AI world, and many modern models support file inputs. The idea was to use an AI model to read the document and output it as a JSON structure. Then, use the External Integration Framework (which can work with JSON data) to create the required D365FO documents.

I tried several models and got the best results with Gemini 2.5 Flash. It’s among the fastest on the market, and performance is essential. For example, an invoice with ~200 lines (4–5 pages) takes about 2 minutes to process.

After some quick prototyping with Claude Code, I extended the External Integration Framework with a “Get data from AI” option. Below are the setup and usage instructions.

Note: Microsoft recently released a Power App Sales Order Agent that implements a similar concept of processing documents by defining custom prompts and then utilising virtual data entities. However, the customisation option is unclear(probably not even possible).

External integration AI solution setup

In this section, we'll discuss how to set up AI for PDF import.

AI Providers

You can have multiple AI providers. Define a provider using the AI Providers form. A provider is a class that extends a DEVIntegAIProviderBase and defines the following:

A set of parameters with default values that the user may override(in this case it is a model)
A free‑text description of how to set up the provider
A connection reference that defines the endpoint and API key
A method that implements the call to the AI provider

In the current implementation, we can use only one Gemini class, but you can extend it later.

Connection type

To store an API key, we need to create a connection type. The Google Gemini endpoint is: https://generativelanguage.googleapis.com/v1beta/models/.

To obtain the key, sign in to https://aistudio.google.com and click the "Get API Key". The price is a fraction of a cent per 1 page.

Enter these values in the Connections form.

Also, we need to create a Manual connection for the inbound message.

AI Prompt Definition

After setting up the connection, define a prompt. The AI Prompt Definition form lets you define and validate the prompt.

First, validate that the connection works by sending a simple “Hi” prompt. The Call API button runs the prompt, displays the response, and shows related statistics.

Then define a prompt for invoice parsing and test it on sample invoices.

A sample prompt used here:

You are an expert at extracting structured data from multi-page PDF invoices. Your task is to process the entire document and produce a single, valid JSON object.
The JSON object must have two top-level keys: HEADER and LINES.
    HEADER: A single JSON object containing summary information from the invoice.
    LINES: A JSON array of objects, where each object represents a single purchased item.
Extraction Rules for HEADER:
    VendorName: This is printed on the first page header before or after the "Tax Invoice" label after "Vendor:" label. 
    ReceiptDate: Find the "Delivery date" date on the first page. Convert this to a "yyyy-MM-DD" format.
    PurchPoolId: Extract the value located directly below the "Delivery date" date on the first page. 
    ChargeValue: At the end of the document, there is a line that states "Handling fee". Get the value from it, the data type should be real, if not found 0.
    InvoiceTotalAmount: On the final page, find the line "Grand Total". Extract the final numeric total. The data type must be a number.
    InvoiceTotalQty: On the final page, find the line "Total Quantity". The data type must be a number.

Extraction Rules for LINES:
Each object in the LINES array represents a single invoice item and must contain the following fields. IMPORTANT: Do not include summary or Total by lines as items in this array.
    InvoiceItem: Take from the "Description" column.
    ItemId: Check the "Description" column. If this is 37' TV, then use "T0004", if 52'TV, use "T0005"
    Quantity: Take the numeric value from the "Qty" column.
    Price: Take the numeric value from the "Unit Price" column.
    GST: Take the numeric value from the "Tax Amount" column.
    TotalValue: Take the numeric value from the "Amount" column.
    Color: This value is determined by grouping. The invoice items are separated into sections by summary lines like "Total by color Black", "Total by color Silver".
        First, identify these summary lines throughout the document.
        All regular item lines that appear before a specific summary line (e.g., "Total by color Black") belong to that group.
        The Color for all items in that group should be the code from the description (e.g., for "Total by color Black", the Color is "Black"; for "Total by color Silver", the Color is "Silver"). Apply this logic consistently across all pages.

Output ONLY the raw JSON. DO NOT INCLUDE any other text, explanations, or markdown formatting.

If you notice that the abilities are quite powerful, we can utilise AI to implement simple mappings. For example, to convert from InvoiceItem to ItemId, I defined a couple of rules. To derive the Color for each line, we used quite a complex logic based on the “Total by color” group.

However, if the mapping requires numeric comparisons (e.g., greater/less), Flash 2.5 can struggle; in that case, include all the necessary data in the output and implement the logic in X++.

Output for the invoice above looks like this:

{
  "HEADER": {
    "VendorName": "Contoso Asia",
    "ReceiptDate": "2025-09-07",
    "PurchPoolId": "01",
    "ChargeValue": 50.00,
    "InvoiceTotalAmount": 3751.50,
    "InvoiceTotalQty": 8
  },
  "LINES": [
    {
      "InvoiceItem": "37' TV Model A",
      "ItemId": "T0004",
      "Quantity": 2,
      "Price": 400.00,
      "GST": 80.00,
      "TotalValue": 880.00,
      "Color": "Black"
    },
    {
      "InvoiceItem": "52' TV Model B",
      "ItemId": "T0005",
      "Quantity": 1,
      "Price": 450.00,
      "GST": 45.00,
      "TotalValue": 495.00,
      "Color": "Black"
    },
    {
      "InvoiceItem": "37' TV Model A",
      "ItemId": "T0004",
      "Quantity": 2,
      "Price": 420.00,
      "GST": 84.00,
      "TotalValue": 924.00,
      "Color": "Silver"
    },

This output will be used as initial data for the processing class.

Inbound message type

To setup processing, we need to create a new Inbound message type with a processing class DEVIntegTutorialPurchOrderOCRProcess which reads the JSON from the AI call and creates a purchase order based on it.

As a custom parameter, this class requires a link to the previously defined prompt.

Operation parameters

We also need some custom parameters. When importing documents, you’ll likely define rules for tax calculation. There are usually two options: either take the value from the document (and then use the line Tax adjustment) or calculate it directly in D365FO. The latter requires defining a Zero tax group.

Main modification description

After specifying all settings, we can test the import. Open Tutorial Purchase orders OCR Staging and select New order import.

The system analyses the provided PDF and creates staging data.

Invoice validation

The next step is validation. AI models may hallucinate, so it’s crucial to establish proper validation.

In our case, we have three validation points:

The total quantity of all lines must match the Total Quantity from the invoice.
The total amount of all lines plus Charge amount must match the Total amount from the invoice.
For each line, Quantity × Price + GST must match the line value. The Rounding field should be 0.

You can also edit any field in the staging tables if needed.

Purchase order creation

After validation, select Process to create a purchase order. The purchase order is created, and the staging data reference is updated (so you can always trace an individual order to the original PDF file).

From this form, you can also open the created purchase order using Open PO.

Resources for This Blog Post

All resources mentioned in this blog post are available on GitHub. Here’s a brief overview of what’s included and how you can use these resources as a starting point for your AI integration projects.

The main components are:

Tables and a form to manage staging data.
A Processing class, DEVIntegTutorialPurchOrderOCRProcess, which contains the logic for parsing JSON data from AI, saving it to staging tables and creating a purchase order based on it.
A class for a "New order import" dialog, DEVIntegTutorialPurchOrderOCRManualImport. In this sample it’s empty, but you can extend it to handle parameters in addition to the provided file (e.g., different formats, missing fields, etc.).

Once these components are set up, the External Integration framework automatically handles the remaining integration process. The solution uses vanilla X++ code without external DLLs and works even on a local VHD.

Summary

In this post, I explained how to use AI to implement a complex PDF document import into Dynamics 365 Finance and Operations using the External Integration framework. We covered these key topics:

How to setup an AI prompt and test it.
Examples of processing AI output and creating purchase orders in Dynamics 365 Finance and Operations.
How to validate the results and create a document.

I hope you found this information helpful. If you have any questions, suggestions, or improvements, please feel free to reach out.

D365FO Integration: Import Sales Orders from an External Web Application

Wed, 12 Mar 2025 22:12:03 GMT

In this blog post, I'll walk you through the process of importing sales orders into Dynamics 365 Finance from an external web service using REST API calls.

Integration Scenario Overview

Let's start by defining our integration scenario:

Goal: Design and implement an integration solution that imports sales orders into Dynamics 365 Finance from a partner's website via a REST API endpoint.

To illustrate this integration, I will refer to the Purchase Order Management application mentioned in my previous post. This application offers an API endpoint that retrieves created or updated orders. Although the example provided is simplified, similar APIs are frequently found in other systems, such as Shopify.

You can access the complete source code for the Purchase Order Management application on GitHub. The application consists of two primary components:

Frontend Application: Provides a user interface for viewing and managing orders.

Backend API: Exposes a REST API endpoint to retrieve orders created or updated after a specified date.

The API requires an UpdatedAt date parameter and returns all orders created or modified after this timestamp. Our integration task is to retrieve these orders and load them into Dynamics 365 Finance.

Planning and Scoping Integration Project

Before diving into the technical implementation, I recommend organising a kickoff meeting involving key stakeholders from both the Dynamics 365 Finance and Operations (D365FO) team and the external Web application provider. During this meeting, ensure you cover the following critical points:

Define the Web API Endpoint and Parameters

Most web applications have well-defined endpoints that accept multiple parameters and return structured data. In our scenario, the API endpoint retrieves orders modified after a specified date using the updatedAt parameter. For example:

https://purchaseorderapp20240916.azurewebsites.net/api/PurchaseOrder?updatedAt=2025-02-07T16:00:00Z

Discuss what authentication should be used; in a typical scenario, a token-based approach is used.

Confirm the Scope of Returned Documents

At this stage, it's essential to clearly define and document exactly which documents the API will return based on the provided parameters. In my experience, API sometimes may also return internal or irrelevant documents—such as orders for locations not managed in D365FO or internal transfer documents. This can significantly complicate integration logic and data validation processes. To avoid these issues, ensure you obtain a confirmation from the external Web application team that only documents relevant to Dynamics 365 Finance will be returned by the API.

Establish Rules for Handling Updated Documents

A document may be modified after it is retrieved via an API. Typically, when a document is updated, the updatedAt timestamp changes, while the document's unique ID remains constant. It's important to define when these updates happen and to understand how often they occur. Ideally, updates after initial retrieval should be minimized or avoided altogether, but this may be a business requirement.

To manage document updates effectively, consider requesting the external Web application team to implement a clear status indicator. This indicator should explicitly mark when a document has been finalized and exposed via the API. For example, users might initially create an order, add necessary details, and then explicitly mark it as "Confirmed" or "Completed." Only after this explicit action the order should become available through the API.

On the Dynamics 365 Finance side, you must establish clear rules for processing updated documents. For sales orders, common approaches include reversing or deleting the original order and creating a new one. Keep in mind that reversing transactions can introduce complexities, e.g. the ledger period for the original order may be closed. In such cases, you might need to use the nearest available open date or implement alternative business logic to handle these scenarios.

In this blog post, I use a simplified approach: I delete the original order if it remains open and generate an error if the order status is no longer open.

Discuss the TEST Endpoint

Confirm whether a test version of the Web application is available and clarify what kind of data it contains. Typically, a test environment exists, but may include dummy data. This can cause issues because Dynamics 365 Finance integrations usually require close to production data. If the test data isn't suitable, you might need to perform some integration testing directly against the production endpoint.

Request a Guide for Creating Documents via the User Interface

Ensure someone from the D365FO team knows how to create documents using the external Web application's user interface. These documents should then appear in the API—for example, creating a sales order that will be imported into D365FO. This step is essential for testing, so request clear instructions on how to perform this task.

Resolving Mapping Errors

It's common for certain entity codes to need mapping between systems. Define which system will handle this mapping and discuss the steps for adding new mappings values.

In our example, I'll store the Customer account mappings in a separate table.

Create a Data Mapping document

Mapping documents is a key integration document that defines how fields from the Web application API will be mapped to D365FO fields.

In our case, it will look like this:

High-level Design Description

A single web service call might return multiple unrelated orders, but we still need clear traceability from each individual order back to the original web request. To achieve this, I introduced the concept of a parent message. Here's how the processing logic works:

The load class sends a request to the Web service using the current message's UpdatedAt date.
The response from the web service creates a parent message. The returned JSON data is attached to this parent message, and processing begins.
To process the parent message, we parse the attached JSON data and create multiple child messages—one for each order. Each child message contains parsed data stored in staging tables. During this step, we also perform basic cleanup actions based on unique identifiers, such as deleting previous messages with an "Error" status that share the same ID. This step is executed as a single transaction in a single thread.
If the parent message processing completes successfully, we update the message's UpdatedAt date to the new value. (If paging is used, we update the paging information instead).
Finally, each child message is processed separately in batch tasks. These tasks can run in parallel (multithreaded mode) and use the staged data to create the final documents in Dynamics 365 Finance and Operations. Errors encountered during this stage do not affect the parent message.

Setting Up the Integration

Let's walk through the steps to set up and validate our integration solution.

Configuring Connection Types

First, we need to create a connection to our web service. To do this, go to External integration > Connection types.

Here, we define the connection details for our web application:

Provide the following information:

URL: The web address of our application (for example, https://purchaseorderapp20240916.azurewebsites.net/)
Timeout: How long the system waits for a response before timing out
Credentials: Username and password details for authentication

Dynamics 365 Finance offers several ways to store passwords securely:

Manual entry: An unencrypted string, suitable for development. It will persist even after database restores.
Encrypted: A secure option that encrypts the password value.
Azure Key Vault: Links to the standard D365FO key vault for the most secure password storage.

Defining Inbound Message Types

Next, we'll set up the import parameters using the Inbound message types form.

This form includes the following key sections:

Connection Details

Here, you link the message type to the previously created connection and specify the processing class (DEVIntegTutorialWebSalesProcess).

INCREMENTAL LOAD

Last date time: The date and time used for the data load. This value (adjusted by the Overlap setting) is passed as the UpdatedAt parameter in the next request. After each successful load, this value updates to the current time.
Overlap (seconds): The number of seconds subtracted from the Last date time when making a request. This helps account for delays between data being written and the transaction being committed in the source database, ensuring no data is missed.
Offset: Indicates the current page number if paging is enabled. This value updates automatically during the load process.
Limit: Defines the maximum number of records returned per page if paging is used. If the number of records returned equals this limit, the system increments the Offset by one(goes to the next page) and does not update the Last date time.
Do not process on load: This is a debug option that prevents messages from being processed immediately after import.

Mapping Setup

In real-world integration scenarios, you'll likely need to map certain fields between systems. The External Integration module provides a simple form and table to handle these mappings. Let's walk through an example.

We'll map the Order account from the Web service to the corresponding Customer account in D365FO. You can configure this mapping under External integration > Setup > Mapping setup.

Running the Integration

Generating Test Data

To test our integration, we'll first generate some sample data. Enter three or more item numbers (for example, M0001; M0004; M0007) and click Load initial data.

The web application will generate three orders based on these items.

Once you've completed all setup steps, you can run the Load messages batch job.

This batch job executes the processing class linked to your inbound message type. As a result, four initial messages will be created and processed:

One parent message containing the original API data.
Three child messages created from parsing the parent message data into staging tables.

From this form, you can:

View the original data from the web application by clicking the Download file button.

View the staging data associated with each individual message.

Check the infolog for detailed processing information, reprocess messages if needed, or open the created Sales order.

Running an Incremental Load

To test how an incremental load works, I'll create a new order in my web application using a new account (ACCT-004) that doesn't yet exist in our mapping setup. I'll also update an existing order (order number 95).

After running the Load messages job again, you'll notice two new records in the web staging data:

One record will show an Error status because we haven't set up mapping for the new account (ACCT-004).
Another record will reflect the updated details for order 95.

Additionally, the system automatically removes the sales order (001035) linked to the original message, as it has now been replaced by the updated data.

Monitoring and Error Handling

For any integration, it's essential to monitor each step and have a clear plan to handle potential issues.

Connection Errors

If Dynamics 365 Finance and Operations (D365FO) can't connect to the specified API, an exception occurs. This exception stops the Load messages batch job, and the batch job status changes to Error.

Another common issue is when the Load messages batch job isn't running at all.

You can detect both situations by setting up an alert on the Last Date Time field. If this timestamp is outdated, it indicates a problem.

Parsing Errors

Parsing errors happen when the Load messages job receives data but can't correctly interpret it.

To monitor parsing errors, you can create an alert on the Incoming messages table for records with a status of Error.

When a parsing error occurs, the system creates a parent message containing the original data. You can analyze this data and forward it to a developer, who can then use the Manual load function to troubleshoot and process the message.

Manual Load Function

The Manual load function is a valuable feature of the External Integration Framework. It allows you to process messages without connecting to the Web application. You simply provide JSON data as text, and the system processes it exactly as if it came from the API.

For example by providing the following JSON data

[  {
    "id": 96,
    "purchId": "PO-002",
    "orderAccount": "ACCT-002",
    "updatedAt": "2025-03-12T02:58:18.641909",
    "lines": [
      {
        "id": 172,
        "purchId": "PO-002",
        "itemId": "M0004",
        "quantity": 8,
        "lineAmount": 120
      }
    ]
  } ]

We can create a new sales order, which will be marked as Manual in the Messages table.

This feature is also helpful for testing different processing scenarios.

New Load Function

You can manually start loading orders from the user interface by clicking the New load button.

In the load dialog, the Transaction time field is automatically populated with the last loaded timestamp. However, you can change this value if needed. A typical scenario for changing this timestamp is when you need to reload data from a specific period.

Handling Repeated Errors

Each time the system attempts to process a message but encounters a failure, the Processing attempts counter for that message increases.

When running the Process incoming messages batch job, you can set a filter on this field to limit the number of times the system tries to process messages with errors.

This approach helps prevent the system from repeatedly trying to process error messages that no one is addressing, which could slow down overall performance. A good practice is to set this filter to allow several processing attempts. This way, temporary issues, such as a missing customer record, can be resolved automatically once the required data becomes available.

Canceling Messages

If a message can't be processed because it contains invalid data, you can choose to Cancel it.

The main goal of this feature is to ensure that messages don't remain in an Error status indefinitely. Ideally, all messages should either be Processed or Cancelled.

Full Traceability

The system provides complete traceability, from the initial JSON message to the final Sales order, and from the Sales order back to the original Web request.

Starting from the sales order:

You can view the staging data with parsed values:

And also, access the original message with the attached JSON payload:

Each of these steps can be performed without direct access to the Web API in a development environment, either by using the Manual load function or by manually changing the status of individual messages.

Resources for This Blog Post

All resources mentioned in this blog post are available on GitHub. Let's briefly review what's included and how you can use these resources as a starting point for your integration projects.

The main components are:

A Load class, similar to DEVIntegTutorialWebSalesLoad, which connects to a custom API.
A Processing class, DEVIntegTutorialWebSalesProcess, which contains the logic for processing incoming data.
Tables and forms to manage staging data.
A mapping extension to implement mapping between Web application and Dynamics 365 Finance and Operations.

Once these components are set up, the External Integration framework will automatically handle the rest of the integration process.

Sample Web App

For testing purposes, I've set up a sample web app. At the time of publishing this post, it is deployed at the following address: https://purchaseorderapp20240916.azurewebsites.net/

The source code for this sample web service is also available on GitHub.

Summary

In this post, I've explained how to implement a complex document integration from a REST API web service into Dynamics 365 Finance and Operations using the External Integration framework. We covered these key topics:

Designing an integration solution.
Examples of querying a Web application and creating sales orders in Dynamics 365 Finance and Operations.
Monitoring and troubleshooting common integration issues.

I hope you found this information helpful. If you have any questions, suggestions, or improvements, please feel free to reach out.

Optimizing Your Legacy Dynamics AX System(AX2012, AX2009, AX40)

Tue, 19 Nov 2024 22:12:03 GMT

If you’re still running Dynamics AX 2012, AX 2009, or AX 4.0, maintaining efficiency is essential, especially if an upgrade isn’t planned. In this post, I will provide a structured approach to managing and optimizing such systems.

Perform a Technical Audit

The first step of system maintenance is to check that your system is configured properly and has no critical issues. To do this, you need to perform a Technical audit. The process has two primary parts. First, gather user feedback about their main pain points using a structured format and resolve these issues.

Next, validate the system’s configuration against best practices and make necessary adjustments. These steps are covered in more detail in my articles Dynamics AX performance audit, which discusses what should be checked and How to manage a Dynamics AX performance optimization project about how to organize this project.

It’s important to involve upper management, as changes in legacy systems often involve risks that require approval.

The technical audit's goal is to achieve a stable and well-configured system that meets best practice standards. This will provide users with a smoother experience and ensure the system operates within acceptable performance parameters.

Perform Regular System Monitoring

After stabilizing your system, it’s vital to conduct regular monitoring to catch and resolve issues early. Windows provides plenty of counters for monitoring, but it’s important to focus on actionable insights rather than just tracking metrics.

For example, simply tracking CPU load isn't enough - seeing CPU usage at 90% doesn't tell you what to do next. In this section, I’ll outline key parameters that not only indicate system health but also lead to some actions.

-Database size monitoring

To effectively monitor database size, export the top 20 tables by size into Excel and use VLOOKUP to track monthly changes.

In Excel, it looks like this:

The “Growth” column in Excel sheet will highlight any anomalies. This makes it easy to spot cases such as unintentional "log" activations, where a table may grow unusually and climb in the rankings.

-Long queries and parameters sniffing monitoring

This monitoring type helps identify critical performance issues before users report them. Parameter sniffing can be visually illustrated by this image: operations that normally take 1-2 seconds to execute suddenly take minutes due to a poorly chosen SQL execution plan.

The monitoring process includes:

A SQL Agent job, that runs hourly to detect new TOP SQL queries.
Alerts when a query enters the TOP 3 resource-consuming statements.
CPU usage statistics for the last 30 minutes to filter out false alerts.
Storing the current execution plan and query text for analysis.

When a new query enters the top list, an alert email is sent with detailed information:

This monitoring takes some time to set up(initially, about 200 queries need to be analyzed) but allows the detection of complex AX performance problems. Full details can be found here: Monitoring the most problematic performance problem in Dynamics AX - parameters sniffing.

-Missing indexes monitoring

Monitoring missing indexes is crucial for maintaining optimal performance. A daily SQL Agent job identifies the top 30 missing index recommendations with over 99% impact and sends email alerts.

It consists of the following components:

SQL Agent job that runs daily and emails TOP30 Missing Indexes recommendations with more than 99% impact
In most cases, a new index should be created, but the script allows to define exceptions

Detailed setup instructions are available here: Dynamics AX performance monitoring: missing indexes

-Business operations performance monitoring

This monitoring is used when you have critical business operations. It includes a Dynamics AX form to track operational details and a Power BI dashboard for visual insights.

The Dynamics AX form:

Logs the performance of each operation.
Records key data such as operation date, duration, and number of rows processed.

The Power BI dashboard helps detect anomalies and view performance trends over time.

More details are here: Implementing of Dynamics AX business operations performance monitoring

-Batch performance monitoring

Batch job performance is a key factor in system efficiency. Using a Power BI dashboard, you can monitor the duration and performance of Dynamics AX batch tasks.

Key features:

Analyse batch tasks by task name, server name, duration, start time, and company.
Identify "not enough threads" issues, where tasks are delayed due to limited processing slots.

Additionally, batch jobs can be analysed by time of day, helping to identify patterns or peak load periods.

While this approach may not be suitable for every customer(as there may not be stable load patterns), it may provide valuable insights about Dynamics AX batch performance. Read more here: Analysing Dynamics AX / Dynamics 365FO batch performance using Power BI.

Developing New AX Integrations

Developing new system integrations may be necessary as business needs evolve. As many legacy AX modules, like AIF, are now outdated, I recommend using the External Integration module. Although designed for D365FO, this X++-based module can be adapted for older AX versions with some limitations.

Using this approach, inbound flow integrations typically involve importing files from a shared directory. Outbound flows may consist of periodic or event-driven exports. Additionally, these can be enhanced with No-Code tools that handle file transfers to external services.

Maintaining Windows and SQL Server Versions

Managing legacy Windows versions can be challenging. Dynamics AX components such as AX client and Application Object Server (AOS) often require specific Windows versions. To check which versions are compatible, search for "Dynamics AX [version] system requirements." For example, Dynamics AX 4.0 can be supported on Windows 2012 R2, which can probably be considered the minimum Windows version.

SQL Server provides more flexibility. You can use the latest versions (e.g., SQL 2022 or SQL 2019) in compatibility mode, which works really well with all Dynamics versions. Vasily Nosov also wrote an article about using the latest version of SSRS: How to install AX 2012 R3 SSRS extensions on SQL 2022 if you really want to.

Performing Data Cleanup

Data cleanup in Dynamics AX has two main activities: periodic cleanup for log-type tables and date-based cleanup for user data. Both processes can be implemented using a cleanup framework. Check this post Implementing Dynamics AX 2009/2012 database cleanup and some sample implementations

The log-type table cleanup removes older data (e.g., over 90 days) and is usually run weekly.

For user data, the process is more complex. Most regulatory bodies (tax authorities) require data to be retained for at least seven years, and legal records may have no retention period. The best approach is to keep seven years of data in the active database, perform long-term backups, and clean up top user tables annually to maintain stable database performance and size.

The cleanup process is very customer specific, you will need to create a list of top tables and define rules for each table:

One of the more challenging tasks is to delete records related to InventTrans, which may require the creation of summary records and financial-related transactions.

Another typical request is to delete inactive companies. This can be done with the following script: deleteCompanyByList.

Summary

This post covered essential concepts and actionable steps for optimising and maintaining a legacy Dynamics AX system. I hope you find these insights helpful. Don’t hesitate to ask questions or share your own experiences with system maintenance and support.

Practical Hints for Technical Management of D365FO Go-Live

Fri, 25 Oct 2024 22:12:03 GMT

Having recently completed several Microsoft Dynamics 365 for Finance and Operations (D365FO) projects, I want to share a few practical hints that helped my team successfully navigate the Go-Live stage. These insights are based on real-world scenarios and are meant to complement standard methodologies with a hands-on approach.

Environments management

In every D365FO project, Microsoft provides a single Tier 2 environment as part of the standard subscription. It’s important to name this environment correctly from the start—avoid calling it UAT or TEST and instead use PRE-PROD (PP or PPN).

A common issue is that, after Go-Live, business users continue using the PRE-PROD environment for Testing purposes. This can lead to complications, as most post-Go-Live issues are data-related and can’t be reproduced in DEV you’ll often need a copy of the PROD database to investigate these issues. However, the PROD database can only be copied to PRE-PROD before being distributed to other environments for debugging

The same rule applies to code deployments—before installing a new code package on PROD, it must first be deployed on PRE-PROD, which usually causes about an hour of downtime.

If you name this environment UAT, some users or consultants might think it’s a test area for them to play around with, leading to complaints when you have to restore the database and wipe their work.

That’s why it’s crucial to establish from the start that the PRE-PROD environment can be refreshed from PROD at any time, and it should never contain any valuable data. This ensures that users don’t treat it as a permanent testing environment. The key here is to create a process where data refreshes and code deployments to PRE-PROD don’t require formal approvals—just a quick notification to the team should be enough to keep things running smoothly.

If your project involves ongoing development, I recommend having at least two Tier 2 environments: one for PRE-PROD and one dedicated to user testing, to avoid these kinds of conflicts.

For more details on D365FO environments, you can check out my posts Understanding Dynamics 365FO different Tiers performance and Development VM configuration

Proactive Data Restore Process

During the first few days after Go-Live, I recommend proactively refreshing the PRE-PROD environment with PROD data at the end of each day(or in the morning). The main challenge is that certain data, such as integration-related or encrypted data, may not copy over or should not be copied during the refresh. To handle this, create an automated procedure to restore these settings. The key is ensuring this process doesn’t require significant manual intervention.

Here’s the process we followed:

Run the PRE-PROD data refresh from PROD (for smaller databases, this typically takes around one hour).
Execute a script that adjusts PROD-specific settings to their Test values (in our case, it was an SQL script to modify integration settings, enable users, etc.).
Perform a database export job for PRE-PROD to LCS (which usually takes at least another hour).

This proactive approach offers a significant advantage: if an issue occurs on PROD, there’s a high chance it can be replicated on the restored PRE-PROD data. In such cases, a developer can restore the database from LCS and start debugging within 1-2 hours after the issue is reported.

There’s also an option to automate this process further using LCS API and Power Automate, although we haven’t gone that far.

If your database is large and the restore takes too long, first ensure there aren’t unnecessary "log" tables inflating its size by using [this script](https://github.com/TrudAX/TRUDScripts/blob/master/Performance/AX Technical Audit.md#database-size). Additionally, consider implementing tips from my post on speeding up BACPAC database restores from Tier 2 to Tier 1.

Timings the Project Team Should Know

When an issue occurs in PROD, the first step is to investigate and try to understand the problem. If it’s still unclear, you’ll need to initiate a data refresh on the DEV environment. It’s important that everyone on the team is aware of the expected timings for this process. For example, if a normal data restore procedure is required, debugging can typically begin within 4 hours.

Once the issue is resolved in the DEV environment, the fix should be deployed to PRE-PROD (ensure the team understands how long this will take). After validation on PRE-PROD, the fix will be included in the next PROD release. It’s critical that the entire team knows the deployment schedule for PROD to ensure smooth coordination.

Issues are often raised by functional consultants working closely with key users, and having clear project timings in place makes communication and planning much easier for everyone involved.

Release Management

Effective release management is key to ensuring a smooth Go-Live experience. It’s unrealistic to expect the first days to be error-free—even with a fully tested system, issues are likely to arise. Therefore, good planning should include an action plan for resolving possible errors, and it’s better to avoid relying too much on an "agile" approach during this critical period.

Issue Triaging

During Go-Live, you will encounter both critical and non-critical issues. Critical issues should be resolved as quickly as possible, while non-critical issues should be evaluated. If they pose no immediate risk, it’s better to address them within the ongoing work queue.

Ignoring non-critical issues can cause users to lose interest in reporting problems. For example, imagine you’re a user of a new system and encounter an issue that, while not critical, adds an extra two minutes to your workflow. You take the time to report it, but nothing happens. The next time something goes wrong, you might not even bother reporting it, and eventually, you might tell your manager the new system isn’t working well.

To avoid this scenario, it’s essential to maintain user engagement by responding quickly to their feedback and delivering improvements with minimal delay.

PROD Deployments Planning

One of the biggest technical limitations of D365FO is that every PROD release requires 40-50 minutes of system downtime, which makes planning your deployments around this window crucial.

During the first two weeks of the Go-Live period, I recommend planning:

A scheduled release window every day (e.g., in the evening)
An emergency release window in the morning

Even if these windows aren’t always used, planning them in advance saves time and avoids last-minute discussions with users about system downtime. It also helps set realistic expectations for users regarding resolving their issues.

Here’s a breakdown of how a typical release happens and the associated timings:
1. The developer fixes and tests the issue on the DEV environment.
2. The fix is deployed to PRE-PROD for final validation via the Deployment Pipeline, which takes approximately 1.5 hours.
3. The release manager logs into LCS, marks the PRE-PROD package as a release candidate, opens the PROD environment, and schedules the release (40 minutes of downtime).

Keep in mind that PROD releases cannot be fully automated and will require some manual effort, which can vary depending on the project.

I recently ran a poll on LinkedIn and got the following results:

Lastly, make sure to communicate with the person handling the releases to clarify that some overtime may be required during this period.

Code Branch Management

During the Go-Live phase, it’s essential to keep your code branch management as simple as possible. In some projects, we used only a single Main branch, while in others Main and Release branches, with Release being mostly a copy of Main. This approach requires quality control, and Javier Lopez’s series of blog posts provides some useful insights.

It’s easy to overcomplicate this process and end up with a full-time person just to manage branches and releases. But for the Go-Live period, I recommend sticking to a "simple mode."

There are two key factors to consider when planning your branch strategy:

Speed of Deployment: How quickly can a validated fix be deployed to PROD? During Go-Live, this should ideally be within 1 day.
Effort Required: How much time does it take to deploy? A good setup shouldn’t take more than 1 hour of work time.

Integration Issues

During the initial Go-Live phase, you’ll likely encounter a wide range of integration-related errors.

For inbound integrations, typical issues include:

External systems sending messages in the wrong format (e.g., missing XML tags, incorrect date or numeric formats)
Messages containing invalid values (e.g., references to data that doesn’t exist in D365FO)
Duplicated messages
Incorrect field mapping on the D365FO side
Misconfigured or missing D365FO settings during message processing

For outbound integrations, common problems are:

External systems reporting that they didn’t receive some documents from D365FO
Or Received messages containing wrong values for individual fields

For Service/OData-based integrations:

Complaints from external systems about incorrect data returned by D365FO (often accompanied by screenshots showing empty or incorrect fields in their UI)

In addition to these issues, you may also encounter errors in the X++ code that processes integration messages(so need to reverse and reprocess existing messages), or experience performance bottlenecks during high message volumes.

On my most recent project, I used the External Integration framework, which provides comprehensive logging and allows for replaying/debugging individual messages to troubleshoot issues. It worked exceptionally well for investigating and resolving common integration problems.

If you’re using a different integration approach, make sure the team has a plan in place to address at least the types of issues mentioned above.

Update Planning and Feature Management

Microsoft releases eight updates per year for D365FO, and you can pause two of them. I highly recommend using one of these pauses during your Go-Live phase to ensure the system remains stable without the added complexity of updates.

Feature Management

Feature management in D365FO allows you to gradually enable new features, but it’s not a configuration tool. If you're a new client, a good strategy is to enable all relevant features before starting your main user testing. This ensures that you’re testing the system as it is in its current version rather than working with legacy code that may no longer be supported.

To do this, simply press the "Enable all" button in the Feature Management workspace, but make sure to exclude any features marked as "Preview," as these are not yet fully supported for production use.

X++ Tooling

Having the right tools can significantly simplify issue resolution during the Go-Live phase. I highly recommend exploring the available resources in the X++ community. Below, I’ve listed some of the tools our team relied on most during the initial Go-Live phase.

Field List

The Field List tool is useful for quickly checking data and comparing values between different records. For example, if you have two sales orders that appear similar, but one of them is missing certain buttons or actions, you can easily compare the records to identify the differences.

Call Stack to Infolog

The Call Stack to Infolog tool is a real time saver. It has saved us a lot of time, helping resolve issues in minutes instead of spending hours on data restores and debugging. When enabled for key users, this tool logs an X++ call stack for every error or warning message.

Some D365FO error messages may be unclear—for instance, encountering "Account not specified" during a complex document posting. With this tool, you can see the exact X++ call stack that generated the message, providing crucial insight into the issue.

Standard approach: Without this tool, the usual process would involve restoring the database and setting a breakpoint in info.add();—a much more time-consuming approach.

SQL Execute

The SQL Execute tool is incredibly useful for data analysis. When I first introduced it, some people considered it unsafe (despite similar modifications being possible through ER), but it actually becomes true. Over time, consultants began writing update queries, which brought attention to potential risks. It's important to note that Microsoft does not provide support for correcting damaged data, so care is needed.

To enhance safety, I added separate roles for Select and Update modes, along with extended logging. Update mode was disabled for everyone except one system administrator, who wasn’t involved in user support. This setup worked well for us.

Another feature added since the initial release is the ability to export results to Excel with proper formatting, including displaying Enum labels and converting DateTime to the user's time zone.

Standard approach: Without this tool, you would typically need to export the database to Tier 2 and connect via SQL Management Studio, which is much more time-consuming.

Execute custom code

In every project I’ve worked on, there was at least one issue that required modifying inventory or ledger transactions. This happened for various reasons: data wasn’t loaded correctly, Unit of Measures contained a wrong conversion, or users selected incorrect accounts or pressed the wrong buttons.

Microsoft introduced the Run custom X++ scripts with zero downtime utility, which has been a valuable tool. However, the approval process they implemented is quite strange. The script runs within a transaction, which doesn’t allow for dialogs, and it requires approval from two people before the final run.

The process is different in real-world project scenarios. Typically, you write a custom class to correct the data and run it in PRE-PROD first, then ask key users to review and confirm the changes. Once they approve, you run the same class in PROD. At this point, since the changes have already been tested and validated in PRE-PROD, no further approval is necessary when the code reaches PROD.

The improved Custom Scripts utility eliminates the standard validations and allows you to run a custom class without external transactions, enabling the use of dialogs.

Another option for similar tasks is the MXT - X++ Interpreter from Oriol Ribes Amenós, which may have some compile risks but works well for simple scripts.

Automation Scripts

One of the most important steps in a project is to automate as much as possible to streamline processes. In our project, we used the following automation scripts:

Automated build and deployment pipelines: This setup requires a service account without MFA. You can find a sample instruction for setting up the pipeline here.
Script to prepare the development VM: This script ensures that all necessary tools(e.g d365fo.tools) are installed and available on every project VM.
Script to refresh code on the DEV VM: This script allows you to compile one or all models in the correct sequence.
Various scripts to support the development process: These scripts include important tools such as restoring the Tier 2 database and sharing it through a separate storage account.
SQL script for post-PROD data restore: This script enables users and updates integration settings to test values after restoring the PROD database.

Summary

In this post, I’ve shared practical insights from my experience in technical management for Go-Live D365FO projects. At a high level, the process is simple: you need to organize an efficient workflow for the following activities:

Delivering fixes from Development to Production
Replicating test cases for Production issues on the Development VM (via data restore)
Monitoring the system to address typical issues

However, the complexity lies in the details. I hope you found this post helpful. Feel free to ask any questions or share your own insights—I’d be happy to discuss them further.

D365FO Integration: Event-Based Exports to External Web Services

Mon, 07 Oct 2024 22:12:03 GMT

In this blog post, I will describe how to implement event-based data export from Dynamics 365 Finance to the Web service using REST API call.

While a simplified example illustrates the core concepts, the approach and code samples are rooted in real-world integration scenarios, making them easily adaptable for similar tasks. We'll leverage a free and open-source External integration framework, offering reusable components for building robust integrations.

Modification description

Let's begin with our task sample definition:

Goal: Design and develop a Dynamics 365 Finance integration solution that automatically sends confirmed purchase orders from D365FO to a partner website via REST API endpoint.

To illustrate this process, I utilized Claude Sonnet 3.5 to generate a simple Purchase Order Management application and deploy it to Azure.

The source code is available on GitHub. The application comprises two main components:

A frontend for displaying orders:

A backend that includes an API for accepting these orders:

In our scenario, we will send confirmed Purchase Orders (POs) from D365FO to this API. The UML diagram below illustrates our test process:

Planning and Scoping Integration Project

To initiate an integration project like this, I recommend organizing a kickoff meeting with key stakeholders from both the D365FO team and the Web Service provider (third-party team). During this meeting, address the following aspects:

1. Create a Data Mapping document

Discuss and document what data needs to be sent and how it should be mapped to the third-party system's accepted format. This is a fundamental question for the entire integration and often requires significant time and attention.

A template for this document can be found here.

In our example, to simplify the process, we're sending all confirmed purchase orders for vendors from a specified Vendor group without any complex mapping.

2. Reference Data Management

Data exchange often involves 'reference data' such as Item codes or Vendor IDs. Aligning how both systems manage this data is crucial. Let's explore some common scenarios:

a) Limited data acceptance: The Web service may accept only a predefined set of data. In this case, you might need to create additional tables in D365FO to maintain possible options.
b) Stable, manually loaded data: When reference data is relatively static, manual updates might suffice. For instance, a user could periodically(e.g. monthly) upload an updated item list to the external system
c) Frequently changing data: If reference data changes often, you may need to develop a separate integration process to keep it synchronized.
d) Automatic reference data creation: Some web services can automatically create reference data entries based on the incoming message. In this scenario, ensure all required fields (e.g., Item name, Vendor name) are included in the transmission.

3. Error Handling Rules

Establish clear protocols for error handling. Consider these common approaches:

a) Real-time validation: All business logic validations occur during the API call. A successful call indicates the document has been accepted. This is generally the preferred method.

b) Format-only validation: The Web service only checks the message format during the call. If the format is correct, the message is accepted for further processing.

Option (a) requires designating a D365FO team member to respond to integration errors. Ensure this person has a documented support channel with the Web Service support team. For example, how will the returned message "Item AAA can't be purchased" be processed?

Option (b) is more straightforward from the D365FO side, but it creates some challenges. You need to know the document's current status. This may be implemented as another integration(inbound to D365FO).

4. Data Structure and Cardinality

Address potential differences in data structure between systems. For example:

D365FO might allow multiple lines with the same ItemID in a Purchase Order, while the receiving system may not support this.

Identify these discrepancies early to plan appropriate handling mechanisms.

5. Handling Document Updates

Define how modified documents should be handled. For instance:

In our case, multiple confirmations can be made for one purchase order. The Web service should be capable of accepting and processing updated versions of previously sent documents.

6. API Flexibility

Understand the web service provider's constraints regarding API modifications. Possible scenarios include:

a) Fixed public API: The API is used by multiple clients and cannot be modified. b) Flexible API: The third-party team can allocate resources to adjust the API for this integration.

If option (b) is possible, establish clear communication channels, developer availability, and bug tracking procedures.

7. Batch vs. Real-time Integration

Decide whether document export should occur:

a) Via a batch job (introducing a delay of at least a few minutes), or b) Immediately after a triggering action (real-time).

This blog post will focus on implementing the more complex real-time integration scenario.

Implementing the Export Functionality

Key Components of the Export Class

The External integration framework provides a base class for implementing event-based integration. To provide an example, I created DEVIntegTutorialExportPurchOrder class that extends this base class. Let's examine it's key methods and components:

1. Validation Method: isNeedToCreateLog

This method defines the rules for determining whether a record should be exported. In our purchase order scenario, it checks if:

The purchase order is in a Confirmed state.
The vendor belongs to the specified Vendor Group
Other relevant conditions are met.

boolean isNeedToCreateLog(PurchTable  _purchTable)
{
    boolean                             res;
    DEVIntegParametersPerCompany        integParametersPerCompany = DEVIntegParametersPerCompany::find();
    ;
    if (this.isMessageTypeEnabled() &&
        _purchTable.DocumentState == VersioningDocumentState::Confirmed  &&
        _purchTable.PurchaseType == PurchaseType::Purch &&
        integParametersPerCompany.POExportOnConfirm != DEVIntegTutorialExportOnConfirm::None &&
        _purchTable.vendTable_OrderAccount().VendGroup == integParametersPerCompany.VendGroupId
        )
    {
        res = true;
        if (DEVIntegExportDocumentLog::sentRecordExists(_purchTable, messageTypeTableOutbound.ClassName))  //already marked to send
        {
            res = false;
        }
    }
    return res;
}

2. Bulk Export Method: exportAllData

This method is used for two scenarios: a) Initial data load when starting the integration b) Re-exporting data after making changes to the export logic/contract

It allows users to specify which records to export through a query dialog.

public void exportAllData()
{
    Query                   query = new Query();
    QueryBuildDataSource    qBDS;
    PurchTable              purchTable;
    int                     processedCounter, insertCounter;

    qBDS = query.addDataSource(tableNum(PurchTable));
    qBDS.addRange(fieldNum(PurchTable, DocumentState)).value(SysQuery::value(VersioningDocumentState::Confirmed));
    qBDS = qBDS.addDataSource(tableNum(VendTable));
    qBDS.joinMode(JoinMode::InnerJoin);
    qBDS.relations(false);
    qBDS.addLink(fieldnum(PurchTable, OrderAccount), fieldNum(VendTable, AccountNum));
    qBDS.addRange(fieldNum(VendTable, VendGroup)).value(SysQuery::value(DEVIntegParametersPerCompany::find().VendGroupId));
  
    QueryRun     queryRun;
    queryRun = new QueryRun(query);
    if (queryRun.prompt())
    {
        while (queryRun.next())
        {
            purchTable = queryRun.get(tableNum(PurchTable));
            if (this.insertFromPurchTable(purchTable))
            {
                insertCounter++;
            }
            processedCounter++;
        }
        info(strfmt("%1 record(s) processed, %2 marked to export", processedCounter, insertCounter));
    }
}
public boolean insertFromPurchTable(PurchTable  _purchTable)
{
    boolean  res;
    ;
    if(this.isNeedToCreateLog(_purchTable))
    {
        this.createLogFromCommon(_purchTable, _purchTable.PurchId);
        res = true;
    }
    return res;
}

3. Core Export Method: exportWebMessage

This method implements the actual export call to the web service. It handles:

Preparing the data for export
Making the API call using an HttpClient. As we used a custom service, we also need to define a new load class DEVIntegTutorialExportPurchLoad that implements communication with the service, but In case we integrate using Azure file share or Azure service bus, the External integration framework already has base load classes for this
Processing the response, in our case, getting the number from an external Web Order and linking it to our PO.

public void exportWebMessage(DEVIntegExportDocumentLog    _exportDocumentLog, DEVIntegMessagesLoadBaseType     _loadFileStorageCache)
{
    PurchTable purchTable = PurchTable::findRecId(_exportDocumentLog.RefRecId);
    DEVIntegTutorialExportPurchLoad  exportPurchLoad = _loadFileStorageCache as DEVIntegTutorialExportPurchLoad;
    exportPurchLoad.initConnection();
    DEVIntegTutorialExportPurchContractHeader  contractData = new DEVIntegTutorialExportPurchContractHeader();
    contractData.initFromPurchOrder(purchTable);

    str sJSON = FormJSONSerializer::serializeClass(contractData);
    Num  externalId = exportPurchLoad.postContract(sJSON, _exportDocumentLog.DocumentId);

    ttsbegin;
    purchTable = PurchTable::findRecId(_exportDocumentLog.RefRecId, true);
    purchTable.VendorRef   = externalId;
    purchTable.doupdate();
    ttscommit;
}

4. Data Contract Classes

To structure the JSON data for export, we use data contract classes. In our case, we have:

A header class for purchase order header
A lines class for line items

What I found is that the "X++ Dev Helper for Dynamics 365 F&O" custom GPT for ChatGPT is quite good at creating these classes based on just the sample JSON.

5. Event Handlers for Export Triggers

These methods mark records for export when certain events occur, such as confirming a purchase order. They should:

Execute within the posting transaction
Update or insert a reference in the DEVIntegExportDocumentLog table with Export status to "To send"

[DataEventHandler(tableStr(PurchTable), DataEventType::Updated)]
public static void PurchTable_onUpdated(Common sender, DataEventArgs e)
{
    PurchTable purchTable = sender as PurchTable;
    if (purchTable.DocumentState == VersioningDocumentState::Confirmed  &&               
        purchTable.orig().DocumentState != purchTable.DocumentState )
    {
        DEVIntegTutorialExportPurchOrder  exportMessage = DEVIntegTutorialExportPurchOrder::construct();
        exportMessage.insertFromPurchTable(purchTable);
    }
}

6. Post-Operation Export Methods

These methods run after the main operation (e.g., purchase order confirmation) to initiate the actual export process. They:

Execute outside the main transaction, so any failure in export should not block the document posting
Check for records marked for export
Trigger the export process for those records

The export sequence is presented on the following diagram:

In our case, we want to check at the end of Purch confirmation whether any export records have been created and, if so, run an export operation for these records.

[PostHandlerFor(classStr(FormletterService), methodStr(FormletterService, postPurchaseOrderConfirmation))]
public static void FormletterService_Post_postPurchaseOrderConfirmation(XppPrePostArgs args)
{
    if (DEVIntegParametersPerCompany::find().POExportOnConfirm != DEVIntegTutorialExportOnConfirm::OnConfirm)
    {
        return;
    }

    PurchFormLetterPurchOrderContract contract = args.getArg('_contract') as PurchFormLetterPurchOrderContract;
    if (contract)
    {
        PurchParmUpdate purchParmUpdate = contract.parmParmUpdate();
        PurchParmTable  purchParmTable;
        PurchTable      purchTable;
        DEVIntegExportDocumentLog  integExportDocumentLog;
        ClassName       exportClass = classStr(DEVIntegTutorialExportPurchOrder);
        DEVIntegExportRecordList  exportRecordList;

        //or if one record - DEVIntegExportRecordList::constructFromRecord()
        while select ParmId from purchParmTable
            where purchParmTable.ParmId == purchParmUpdate.ParmId
        join PurchId from purchTable
            where purchTable.PurchId        == purchParmTable.PurchId 
        join integExportDocumentLog
            where integExportDocumentLog.RefTableId == purchTable.TableId
                && integExportDocumentLog.RefRecId == purchTable.RecId
                && integExportDocumentLog.ClassName == exportClass
        {
            if (! exportRecordList)
            {
                exportRecordList = new DEVIntegExportRecordList();
            }    
            exportRecordList.addExportLog(integExportDocumentLog.recId);
        }
        if (exportRecordList)
        {
            DEVIntegExportDocumentsLog::exportRecordList(exportRecordList);
        }
    }

}

Comparing Custom Export Class vs. Standard Business Events

While Microsoft provides a seemingly similar concept with Business events for Dynamics 365 Finance and Operations, there are significant differences when compared to our custom Export class approach. Let's examine these differences in detail:

1. Data Creation Timing

With Business events, data is captured at the moment the event occurs, creating a fixed event payload. In contrast, Export class gathers data at the time of export execution. This allows for real-time data retrieval, including any updates made after the initial trigger. For example, an export may fail due to incorrect or missing mapping. For External integration, you just need to fix this mapping; it will be automatically applied at the time of the next export.

2. Event Frequency and Uniqueness

Business events are generated for each occurrence, potentially creating multiple events for the same document (e.g., two events for two confirmations of the same purchase order). There's also no guaranteed sequence for event delivery. For documents that can't be modified after the export(e.g. Invoices), this difference is probably not critical, but in case exporting documents that can be modified, you need to make sure that the consuming side can handle the message sequence correctly.

Export class, however, maintains a unique Export log entry per document, updating existing entries for repeat actions on the same document. This ensures the most recent state is always exported.

3. Response Processing

Business events are generally designed for one-way communication with limited built-in capabilities for processing responses from external systems. Our Export class easily incorporates logic to handle responses, allowing for updating local records based on the external system's response (e.g., linking local PO with external PO number).

4.No logs

Business events do not support any logs, except for internal send errors. For example, if the event was sent successfully, you won't see this. You also can't find out for the original document - what types of events were sent to it. This is a common integration question, for example, if the external team tells you they don't receive an event, you have no options to validate it when using Business events. External integration has a complete log, and it contains the Original document and the Reference RecId to the original record, so troubleshooting is quite simple; you can always see what events were generated and sent.

Setting Up and Validating Integration

Let's walk through the process of setting up and validating our integration solution.

Configuring Connection Types

First, we need to set up the connection to our web service. Navigate to the External integration – Connection types.

It requires the hostname(will be https://purchaseorderapp20240916.azurewebsites.net/api/purchaseorder) and allows to set a user/password credentials to access this host(in our example, we're leaving these empty)

The system offers several options for storing passwords:

Manual entry: An unencrypted string, suitable for development. It will persist even after database restores.
Encrypted: A secure option that encrypts the password value.
Azure Key Vault: Links to the standard D365FO key vault for the most secure password storage.

Defining Outbound Message Types

Next, we'll configure the parameters for our data export in the Outbound message types form.

Key settings:

Set "Export type" to "Document on event"
Select our custom export class
Specify the connection ID we just created

Setting Up Integration Parameters

For our integration, we also created the following parameters

For our specific integration, we've created some additional parameters. You can find these in the integration parameters form:

Vendor Group: This defines the filter for exported Purchase Orders.
Export on confirmation option: This determines when the actual export (web service call) happens:
- In the user session immediately after confirmation
- Via a periodic batch job

Export scenarios

Initial Export of the Data

Some Purchase Orders (POs) may already exist in the database when setting up the integration. To export these, use the "Export all" button on the "Outbound message types" form. You'll see a standard query dialog like this:

After running the query, all Purchase Orders matching your criteria will be processed. The system will add a record with the PO reference to the Export document log table, setting its status to To send.

Periodic Batch Export

The Export messages from log periodic operation in D365FO process all records in the Export document log table with a Status = To send. You can also manually export by selecting records and using the Export records button:

After processing, you should see the exported orders on our site:

The Reference field will be populated for exported orders in D365FO:

If you've enabled full logging, you can check the exact message sent in the Export document log form:

After a successful export, the Export Status field for the Export document log table will change to Sent:

Export from the User Session

When a user runs a Purchase order confirmation, if any Export document log records were created during this process, the system runs the export immediately after the main operation. The status will be displayed to the user.

If an error occurs during the export, it will be displayed to the user, but the Confirmation process will not be affected. This allows for real-time feedback while ensuring the main business process isn't disrupted.

Handling Export Errors

If an exception occurs during the export, the Export document log record will remain in the To send status. An additional log will be generated with the workload details and exception text.

For example, if a Purchase order contains a service Item (without ItemId), but our test web service doesn't accept such data, you'll see an error logged.

It's important to note that the Export document log has only two statuses (To send and Sent); there's no specific Error status. From a business perspective, it doesn't matter whether the record wasn't exported or if the export failed. The key is that the record needs attention. For effective monitoring, set up your alerting system to flag records that stay in the To send status longer than your defined threshold.

Troubleshooting and Monitoring

Integration can be complex, and proper tracing to identify issues is vital for the whole process. Let's explore the different options that the External integration framework provides for troubleshooting and monitoring:

Tracing the call

One of the most common issues with service calls is when the Web service doesn't process the call properly but returns a success code. For example, when we send an update to an existing PO, the update might fail for some reason, but no error code is returned. This results in mismatched data between the systems.

To trace this error, we need to use the external PO ID to find an export record using Document ID and see the full history of exports.

Using this information, we can determine on which side the error occurred and take steps to fix it.

Manual Test Form

Another useful tool for troubleshooting is the Manual test form. This form offers several options:

"Init from document" button

This button allows you to:

Select a Purchase order number ignoring existing validations.
See how this document will be represented in the data.
Display the workload without actually sending it

"Call Create" button

Take input data (which can be previously generated by the "Init from document" button and then modified by the user)
Perform a call with this data.
Display the results

The advantage of this approach is that it allows you to test the webservice without using any external tools like Postman. This can be very helpful in troubleshooting scenarios.

Monitoring

Monitoring can be done using External Integration workspace that displays statistics for a given period and number of not exported records.

This dashboard gives you a quick overview of your integration's health and performance, allowing you to spot issues early and take action.

Performance testing

The External integration framework provides a simple performance test operation to assess the speed of our integration. Let's explore how this works and what insights we can gain from it.

How It Works

The performance testing process involves several steps:

First, you need to create a set of documents that are ready for export. These will serve as your test data.
Next, run the Performance test operation. This operation allows you to:
- Specify a standard query filter for the Export document log table (this means you can create multiple clients if needed)
- Run processing of these records in a loop for a specified period (set by the Duration parameter)
The test can be run in two modes:
- Reusing the same connections to export documents
- Creating a new connection (instance of HttpClient class) for every export line

Here's what the Performance test operation form looks like:

For our webservice, I ran the performance test for 60 seconds and got the following results:

When creating a new connection for every line: 350 exports/minute
When reusing the same connection (cached): 700 exports/minute

These results highlight an important point: creating a new connection is a complex operation. It can be even more time-consuming if authentication is involved.

Resources for This Blog Post

All resources used in this blog are available on GitHub. Let's take a look at what's included and how you can use these resources for your own integration projects.

To implement your own D365FO integration with a webservice, you'll need to create two main classes:

A class similar to DEVIntegTutorialExportPurchLoad
- This class handles the interaction with the custom Webservice(if we use Azure service bus or File share, it is not needed)
A class similar to DEVIntegTutorialExportPurchOrder
- This class defines the export document structure.

Additionally, you should create a form for manual testing of these classes, similar to DEVIntegTutorialTestWebCall.

Once these components are in place, the External integration framework will handle all other aspects of the integration process.

Sample Web App

For testing purposes, I've set up a sample web app. At the time of publishing this post, it is deployed at the following address: https://purchaseorderapp20240916.azurewebsites.net/

The source code for this test web service is also available on [GitHub] (https://github.com/TrudAX/TestWebService_PurchaseOrderApp)

Summary

In this post, I have described how to implement event-based exports from Dynamics 365 Finance and Operations to Web Service using the External Integration framework. We discussed the following key topics:

How to design such integration
Sample implementations of how to call a web service and how to create a document class
How to monitor typical issues with such integration
How to perform performance testing

I hope you find this information useful. As always, if you see any improvements or suggestions or have questions about this work, don't hesitate to contact me.

Implement Periodic Data Export from D365FO to SFTP

Fri, 12 Jul 2024 22:12:03 GMT

External integration is a framework designed for inbound and outbound integrations in D365FO. It supports several channels: Azure file share, SFTP, Azure service bus, and provides comprehensive features for logging, error handling, and troubleshooting.

In this blog post, I will describe how to implement periodic data export from D365FO to a file and upload it to an SFTP server.

SFTP Server Setup

A few years ago, Microsoft did not support SFTP hosting, claiming it was obsolete in the cloud world. However, SFTP remains quite popular among clients and eventually, SFTP support was included in the Azure storage service.

To use Azure storage via an SFTP endpoint, you must first create a storage account and enable SFTP access. More information can be found here: SSH File Transfer Protocol (SFTP) support for Azure Blob Storage.

Then, create an SFTP user:

Please note that SFTP access incurs additional charges of about $10 per day compared to Azure file storage. Alternatives such as files.com offer additional features, including more granular access control.

Client Library for SFTP Connection

Several .NET libraries are available for working with SFTP. One of the most popular is the open-source SSH.NET. It adds a reference Renci.SshNet.dll that can be used in X++.

For user interface operations, I recommend using FileZilla.

System Setup

Before exploring different integration scenarios, let's discuss common setup options.

Connection types

The first form to setup an SFTP connection is External integration – Connection types.

It requires the hostname and user/password credentials to access this host.

The password can be stored in several ways:

Manual entry - An unencrypted string suitable for development testing. It will remain after DB restores.
Encrypted - An encrypted value.
Azure Key Vault - A link to the standard D365FO key vault that stores the password.

Outbound message types

In Outbound message types, we set parameters for our data export.

Source group Here, we specify a link to the connection type and a folder on the SFTP server for data export.

Processing group Contains a link to the class that performs the export. The class is a standard RunBase class (extended from DEVIntegExportBulkBase) that can define custom parameters and utilizes helper functions from the base class.

File parameters Often, exports require creating a file name that includes a date component. This setting enables specifying a name template using the '%d' parameter. Date format(%d) defines how this symbol is converted into a date (e.g., you can include date-time or use just date). The format rules are defined by the .NET DateTime.ToString format parameter.

Delimiter defines a CSV-type delimiter.

Advanced Group The advanced group defines used log types and a Company range validation. The exports run in the current company, and some exports may be logically related only to certain companies. The Company range allows specifying a list of companies eligible for export.

Export scenarios

Let's describe the most typical scenarios that can be used to export data from D365FO.

Export Store Onhand Data (no-code scenario)

Business scenario: Our company has a store, and we want to export daily onhand data from this store to our website. We need to include ItemId, Warehourse, and AvailiablePhysical value.

The External integration module contains a DEVIntegExportBulkSQL class that allows exporting the result of a custom SQL string. This basic task can be done without writing any code.

SELECT ITEMID, sum(availPhysical) as Qty, InventLocationId FROM INVENTSUM
WHERE CLOSEDQTY = 0 AND DATAAREAID = 'USMF' 
GROUP BY ITEMID, InventLocationId
HAVING sum(availPhysical) > 0

Then we can setup a standard daily batch job that generates a file in our output folder.

Export customers (no-code with data entity)

Business scenario: In our D365FO ERP, we maintain clients and want to implement daily export to the external system. We want to export the Client Id, Name, and E-mail.

This task is similar to the previous one, but we will fetch the data using a standard data entity.

SELECT [CUSTOMERACCOUNT], ORGANIZATIONNAME, PRIMARYCONTACTEMAIL FROM [CUSTCUSTOMERV3ENTITY]
WHERE DATAAREAID = 'USMF'

Message type setup is the following:

The result will be a file with customers:

The examples above are great for prototyping, but "real world" export scenarios will likely be more complex than using a single SELECT statement, so you may need to write X++ code. (One possible improvement is integrating the module with Data management and ER engines. If you have some working code that you can share, please ping me.)

X++ Export Implementation concept

The External Integration framework provides the following development concept: We can't predict how complex our exports will be. They may export to different sources (like SFTP, Azure storage, Service Bus, Web service), export data to one file or multiple files, require different parameters, etc. To cover all possible scenarios, we can use a standard RunBaseBatch class.

However, export classes will have some common methods. So, I created a base DEVIntegExportBulkBase (that extends RunBaseBatch) to store these common properties and methods (like variables to store the number of export records, methods to create logs, write data to a file, etc.). The Export class should extend it.

Export onhand data (simple X++ procedure)

Let's start with a simple case where you need to write X++ code that generates the export data.

Business scenario: We want to export all companies' inventory onhand data to SFTP. Our export should be a CSV file containing the following fields: 'Company', 'ItemId', 'InventLocationId', 'LastUpdDatePhysical', 'AvailPhysical'.

The following class solves this task:

class DEVIntegTutorialExportBulkInventOnhand extends DEVIntegExportBulkBase
{
    public void exportData()
    {
        container  lineData;
        InventSum  inventSum;

        this.initCSVStream();

        lineData = ['Company', 'ItemId', 'InventLocationId', 'LastUpdDatePhysical', 'AvailPhysical'];
        this.writeHeaderLine(lineData);

        while select crosscompany sum(AvailPhysical), maxof(LastUpdDatePhysical) from inventSum
            group by ItemId, InventLocationId
            where inventSum.AvailPhysical 
        {
            lineData = [inventSum.DataAreaId, inventSum.ItemId, inventSum.InventLocationId, inventSum.LastUpdDatePhysical, inventSum.AvailPhysical];
            this.writeDataLine(lineData);
        }

        this.sendFileToStorage();
    }
    public str getExportDescription()
    {
        return "Tutorial export Onhand to CSV";
    }
}

The class is quite simple, but it contains the following key parts:

A business definition for the export
An export flow

For example, if you want to create a file per company instead of one file, you can easily modify it. The External Integration framework will handle all related processes.

After we setup this class, we get the following file as a result:

Export onhand and price data (X++ based on a query)

Let's consider an export scenario with more advanced logic.

Business scenario: We want to implement a periodic export to SFTP of onhand data for one of our customers from their main Warehouse for a set of Items (the default should be the 'Audio' Item group, but it should be configurable). The export should be a CSV file that contains the following fields: 'ItemId', 'InventLocationId', 'AvailPhysical', 'Price'

To do this, we need to create a class that extends DEVIntegExportBulkBase. The code for this class contains the following blocks:

A method that defines the initial query and settings, providing default values(that can be overridden by the user):

class DEVIntegTutorialExportBulkOnhandPricesQuery extends DEVIntegExportBulkBase
{
    CustTable           custTable;
    public Query exportQueryInit()
    {
        Query                 query;
        QueryBuildDataSource  qBDS;

        query   = new Query();

        qBDS = query.addDataSource(tablenum(InventTable));

        qBDS = qBDS.addDataSource(tablenum(InventItemGroupItem));
        qBDS.relations(true);
        qBDS.addRange(FieldNum(InventItemGroupItem, ItemGroupId)).value('Audio');

        return query;
    }
    public DEVIntegMessageTypeTableOutbound initDefaultParameters(DEVIntegMessageTypeTableOutbound    _messageTypeTableOutbound)
    {
        DEVIntegMessageTypeTableOutbound  tableOutbound = super(_messageTypeTableOutbound);
        tableOutbound.FileNameParameterD   = 'yyyyMMdd_HHmm';
        tableOutbound.FileName             = 'OnhandPrices_%d.csv';
        tableOutbound.FileColumnDelimiter  = '|';

        return tableOutbound;
    }

A main export logic that initialises the file and uses the stored query to process export records:

public void exportData()
{
    container               lineData;
    InventTable             inventTable;
    Query                   query;
    QueryRun                qr;

    //init parameters
    DEVIntegParametersPerCompany integParametersPerCompany = DEVIntegParametersPerCompany::find();
    DEV::validateCursorField(integParametersPerCompany, fieldNum(DEVIntegParametersPerCompany, ProductExportCustomer), true);

    custTable = CustTable::find(integParametersPerCompany.ProductExportCustomer);

    this.initCSVStream();
    lineData = ['ItemId', 'InventLocationId', 'AvailPhysical', 'Price'];
    this.writeHeaderLine(lineData);

    query   = this.exportQueryGet();
    qr = new QueryRun(query);

    while (qr.next())
    {
        inventTable = qr.get(TableNum(InventTable));
        this.itemRecord(inventTable);
    }

    this.sendFileToStorage();
}

And a method that calculates the exported values:

void itemRecord(InventTable _inventTable)
{
    InventDim       inventDimFind;
    Price           price;
    InventSum       inventSum;

    inventDimFind = null;
    inventDimFind.InventLocationId  =   custTable.InventLocation;
    inventDimFind.InventSiteId      =   custTable.InventSiteId;
    inventDimFind   =   inventDim::findOrCreate(inventDimFind);

    PriceDiscParameters parameters = PriceDiscParameters::construct();
    parameters.parmModuleType(ModuleInventPurchSales::Sales);
    parameters.parmItemId(_inventtable.ItemId);
    parameters.parmInventDim(inventDimFind);
    parameters.parmUnitID(_inventtable.inventTableModuleSales().UnitId);
    parameters.parmPriceDiscDate(DEV::systemdateget());
    parameters.parmQty(1);
    parameters.parmAccountNum(custTable.AccountNum);
    parameters.parmCurrencyCode(custTable.Currency);
    PriceDisc       priceDisc = PriceDisc::newFromPriceDiscParameters(parameters);

    if (priceDisc.findPrice(custTable.PriceGroup))
    {
        price       = CurrencyExchangeHelper::amount(priceDisc.price());
    }
    else
    {
        if (exportBulkStatus != DEVIntegExportBulkStatus::Warning)
        {
            warning(strFmt("Price is not found for %1 Item", _inventtable.ItemId));
            this.setExportStatus(DEVIntegExportBulkStatus::Warning);
        }
    }
    select sum(AvailPhysical) from inventSum
        where inventSum.ItemId           == _inventTable.ItemId &&
        inventSum.Closed           == false &&
        inventSum.InventSiteId     == custTable.InventSiteId &&
        inventSum.InventLocationId == custTable.InventLocation;

    this.writeDataLine([
        _inventTable.ItemId,
        custTable.InventLocation,
        inventSum.AvailPhysical,
        price ]);

The settings for this class will be the following:

The resulting file will look like this:

As in the previous example, a developer writes only export business logic related to the export.

Warning message status

This class also contains a concept called a Warning status. An export may finish successfully or may have an exception during file generation (Error state). But sometimes, you need to log an event where the exported data has issues but still wants to perform the export. For example, in our case, we can't find a price for some exported items. It is not a critical error, but we need to notify a user about this; they may adjust the query to exclude these items or notify another department to enter prices. To cover this scenario, the export class may mark the status as Warning, and the user may set a standard alert for it.

The Skipped lines counter serves the same purpose. If we need to skip some lines and notify a user about this, we can increase the Skipped counter and see it in the log.

Export invoices to the EDI provider (incremental X++ procedure)

Let's describe a scenario where we need an incremental export.

Business scenario: Our company wants to export customer invoices to an external EDI system. The export runs daily and should include all invoices for this day. The export file should contain information about customer invoices, lines, and charges.

To start this task, you will receive an EDI specification document and need to determine how it will be mapped to D365FO data.

From practical experience, the first question to ask is how the EDI line data is defined. In the best scenario, it may match what you have in invoice lines, but in more complex cases, they can be defined on a more granular level (like related inventory transactions) or on a more summary level (e.g., lines grouped by Item number). Such requirements may even create quite complex system modifications to support such line splits.

The next decision is how to track incremental updates. A typical mistake that I sometimes see on projects is using the CreatedDateTime field for this. The idea is to export all data up to the current time, save this time, and next time process all records starting from this time. The problem with this approach is that it doesn't consider existing transactions. The SQL transaction may start at 1 pm, create an invoice at 1:05, and finish at 2 pm. If the export runs at 1:30, it will not see a 1:05 uncommitted transaction, and the invoice will not be exported. So, it is better to avoid such architecture.

To implement incremental tracking, I propose the following: Add 2 fields, IsExported and ExportedDateTime, to the invoices and update these fields after the export. It may not sound technically perfect (for example, DMF may use SQL change tracking, so you don't need to add any fields), but it is very simple to troubleshoot, and it provides full visibility to a user. They can just open an invoice and see its status and when it was exported.

So when the export runs in the Incremental mode, it just takes all records that are not exported. Another advantage of this approach is there can be a situation when we need to reexport some data (for example, the export may contain errors, or we need to add additional information to the export). With these status fields, it is quite easy to implement.

The main exportData method:

if (exportType == DEVIntegExportBulkIncrementalType::Incremental)
{
    qbds.addRange(FieldNum(CustInvoiceJour, DEVTutorialIntegEDIIsSend)).value(queryValue(NoYes::No));
}
qr = new QueryRun(query);

while (qr.next())
{
    custinvoicejour = qr.get(TableNum(CustInvoiceJour));

    this.exportRecord(custInvoiceJour);

    tmpExportRecordsMark.RefRecId = custInvoiceJour.RecId;
    tmpExportRecordsMark.insert();
}
this.sendFileToStorageNotEmpty();

if (! isTestRun)
{
    ttsbegin;                  
    custInvoiceJour = null;
    update_recordset custInvoiceJour
        setting DEVTutorialIntegEDISendDateTime = exportStartDateTime,
                DEVTutorialIntegEDIIsSend = true
    exists join tmpExportRecordsMark
        where tmpExportRecordsMark.RefRecId == custInvoiceJour.RecId;
    ttscommit;
}

And method that translates invoice to EDI format:

......
while select custinvoicetrans
    order by LineNum, itemid   
    where custinvoicetrans.InvoiceId == _custinvoicejour.InvoiceId &&
          custinvoicetrans.SalesId   == _custinvoicejour.SalesId
{
    linecounter++;
    totalQty +=  custinvoicetrans.Qty;

    InventDim       inventDim = custinvoicetrans.inventDim();

    if (custInvoiceTrans.SumLineDiscMST && custInvoiceTrans.Qty)
    {
        realPrice       =   decRound(custInvoiceTrans.SalesPrice - (custInvoiceTrans.SumLineDiscMST/custInvoiceTrans.Qty),4);
        realPriceStr    =   num2str(realPrice,1,2,1,0);
    }
    else
    {
        realPriceStr   =   num2Str(custinvoicetrans.SalesPrice,1,2,1,0);
    }
    InventItemBarcode inventItemBarcode = inventItemBarcode::findItemId(custInvoiceTrans.ItemId, false, false, 'EAN-13');
    barCodeStr = inventItemBarcode.ItemBarCode;
    if(! barCodeStr)
    {
        barCodeStr  =  custinvoicetrans.ItemId + '-' + inventDim.InventColorId + '-' + inventDim.InventSizeId;
    }

    lastlinenum = custinvoicetrans.LineNum;

    this.writeDataLine(["D",
        num2str(custinvoicetrans.LineNum,1,0,0,0),
        salesTable.PurchOrderFormNum,
        barCodeStr,
        custinvoicetrans.itemName(),
        num2str(custinvoicetrans.Qty,1,2,1,0),
        custinvoicetrans.SalesUnit,
        num2str(custInvoiceTrans.SalesPrice,1,2,1,0),
        num2str(custInvoiceTrans.DiscPercent,1,2,1,0),
        realPriceStr,
        _custInvoiceJour.InclTax ? "Y" : "",
.......

As a result, you will get a file similar to this one:

Below are the dialog and parameters set up for this export. By default, we export invoices from US-005 customers incrementally, but the user can export All invoices for a period or an individual invoice if needed.

Troubleshooting and monitoring

Let's discuss how we can support our periodic exports.

Test run parameter

One of the parameters of the base export class is a Test run checkbox. When running with this parameter, the export sends a file to a user's browser instead of SFTP. For incremental export, it also will not update the "Last exported date" field, so it will not break the export sequence. This is very handy when checking what will be exported without putting anything on SFTP. Also, developers may develop exports without needing to have a connection to SFTP.

Export Log

The export log allows one to view how many lines were exported and the time it took to perform the export. For failed jobs, we can setup a standard D365FO alert.

Test connection

The Test connection button tries to connect to the specified SFTP folder and list files in it. So you can check the connection before running the export.

Summary

In this post, I have described different types of periodic exports to file from Dynamics 365 Finance and Operations based on the External Integration framework. We discussed the following exports:

Simple SQL query export
Export based on custom X++ code and query settings
Incremental X++ export

All used classes can be found on GitHub and can be used as a starting point for your own integrations.

I hope you find this information useful. As always, if you see any improvements, suggestions, or have questions about this work, don't hesitate to contact me.

Step-by-Step Guide: Creating D365FO Build and Deploy Pipelines

Fri, 08 Sep 2023 22:12:03 GMT

Setup Azure Build Pipeline

This guide will walk through the standard procedures for setting up a D365FO Azure-hosted build pipeline.

References

Before we begin, please refer to the following documentation for context:

Procedure detail

Pre-checks

Before diving into the setup, perform a few essential pre-checks:

Install the «Dynamics 365 Finance and Operations Tools» extension
Create personal access token (az)

Navigate to DevOps > User Settings > Personal Access Tokens. Scopes: Full Access, Expiration – 1 year (maximum)
Create a service account user with no MFA and Application registration

Creating the Build Pipeline

Download the 4 NuGet files to your local PC
Log in to DevOps > Your Project > Artifacts > Create Feed
Click the Connect to Feed button and select NuGet.exe

Copy the selected text and paste it into the nuget.config file in the same directory as the previous packages.
Click the Get the Tools button and download the latest version of nuget.exe

Create a new file named packages.config and add the following content (ensure correct version numbers):

<?xml version="1.0" encoding="utf-8"?>
<packages>    
    <package id="microsoft.dynamics.ax.application1.devalm.buildxpp" version="10.0.2177.37" targetFramework="net40" />
    <package id="microsoft.dynamics.ax.application2.devalm.buildxpp" version="10.0.2177.37" targetFramework="net40" />
    <package id="microsoft.dynamics.ax.applicationsuite.devalm.buildxpp" version="10.0.2177.37" targetFramework="net40" />
    <package id="microsoft.dynamics.ax.platform.compilerpackage" version="7.0.7521.60" targetFramework="net40" />
    <package id="microsoft.dynamics.ax.platform.devalm.buildxpp" version="7.0.7521.60" targetFramework="net40" />
</packages>

Obtain the version number from the package files(there should be 2 different versions, 2 for Applications and 2 for Platform).

Open the command line and run the following command for each NuGet file:

nuget.exe push -Source "Dynamics365FO" -ApiKey az C:\AAA\Build\Microsoft.Dynamics.AX.Application1.DevALM.BuildXpp.nupkg
nuget.exe push -Source "Dynamics365FO" -ApiKey az C:\AAA\Build\Microsoft.Dynamics.AX.Application2.DevALM.BuildXpp.nupkg
nuget.exe push -Source "Dynamics365FO" -ApiKey az C:\AAA\Build\Microsoft.Dynamics.AX.ApplicationSuite.DevALM.BuildXpp.nupkg
nuget.exe push -Source "Dynamics365FO" -ApiKey az C:\AAA\Build\Microsoft.Dynamics.AX.Platform.CompilerPackage.nupkg
nuget.exe push -Source "Dynamics365FO" -ApiKey az C:\AAA\Build\Microsoft.Dynamics.AX.Platform.DevALM.BuildXpp.nupkg

When prompted, use: Username = az; Password = token value

Create a new solution with an empty project for each package you want to build. No need to include nested packages. Copy nuget.config and packages.config to the solution folder.
Download the sample pipeline from the following GitHub account:

https://github.com/microsoft/Dynamics365-Xpp-Samples-Tools

(Note: Some users reported issue with wrong task ID, not used now - https://github.com/microsoft/Dynamics365-Xpp-Samples-Tools/pull/2/files )

Open a PowerShell prompt and run the following commands to create a new pipeline:

cd C:\Users\Admin05ec84c604\Downloads\Dynamics365-Xpp-Samples-Tools-master\CI-CD\Pipeline-Samples

./xpp-classic-ci-.ps1

This will create a new build pipeline.

Adjusting the Build Pipeline Parameters

Open the created pipeline and make the following adjustments:

Click Get sources and specify the full path to the source code.
Update the path to the build Solution with empty projects:
Update the NugetConfigsPath path on the Variables tab to: $(Build.SourcesDirectory)\Projects\AzureBuild
Optionally, enable the Licenses Load task:

$(Build.SourcesDirectory)\Licences*.txt
Run the pipeline and verify that everything is working as expected.

Updating the Build Pipeline to a new version

If you need to update a Build Pipeline to a new version follow these steps:

Execute steps 1-6 from the Create pipeline process
Copy nuget.config and packages.config files to the AxBuild solution folder(replace existing files).

Creating a New Release Pipeline

Open Pipeline > Releases and create a new pipeline based on the Empty job.
Add a new artifact
Add the Install MSAL.PS to enable authentication task.
Add the Dynamics Lifecycle Services (LCS) Asset Upload task.

In this task create a new Connection:
- UserId: svc_Dev_LCS@xxx.onmicrosoft.com (service account, no MFA)
- Application (Client) ID: 2eb122fa-a3d0-4a15-b2ff-a8b0339xxxxx
Project ID:

File to upload: Select a file and replace its name:

$(System.DefaultWorkingDirectory)/_X++ (Main)/drop/AXDeployableRuntime_22.04.07.1.zip

with

$(System.DefaultWorkingDirectory)/X++ (Main)/drop/AXDeployableRuntime$(Build.BuildNumber).zip

Reference name: Upload

LCS Asset name: AppRelease$(Build.BuildNumber)

LCS Asset Description: $(Release.ReleaseDescription)
Add the Dynamics Lifecycle Services (LCS) Asset Deployment task

Specify LCS File Asset Id: $(Upload.FileAssetId)

Untick Wait for Completion to prevent timeouts.

Summary

This post provides a detailed guide for setting up Dynamics 365 Finance and Operations Build and Release pipelines. I intend to keep this guide up-to-date, so please report any issues you encounter.

Hope you find this guide useful. Stay updated by following me on LinkedIn or twitter .

Batch Job errors in D365FO: How they can ruin the performance and how to fix them

Thu, 25 May 2023 22:12:03 GMT

In the current version of Dynamics 365 for Finance and Operation, Microsoft introduced an updated batch job processing logic with several main changes:

No dedicated AOS services
Batch retries concept

D365FO batch jobs run in the cloud, and sometimes due to this, they may fail. For example, the batch job may be moved to a different batch server, or the SQL database may be migrated to another pool. In both cases, the execution fails with a system error. To handle these cases, Microsoft has introduced the Batch Retryable feature.

The concept is not complex and is explained in this presentation, Introduction to the BatchRetryable Feature. However, it may be very confusing when you see it in real life. In this post, I will demonstrate the concept with a sample class and discuss its potential impact on real-life batch processing scenarios. )

Exploring with a Simple Test Class

I recently posted a question on my LinkedIn page (many thanks to all who responded), and the results were intriguing:

Only 4% answered correctly: the correct time would be 500 seconds. Feel free to download the sample class and play with it.

Let’s discuss why this happens.

When a batch job starts and encounters an error after 60 seconds, the batch framework checks the number of retries. If this is less than the max retries (5), it executes the batch job again. This cycle(with some queue waits) extends the total execution time to about 500 seconds.

It then raises an interesting question: how many errors do you see in the Batch job execution history for this long-running batch job?

And the answer is more surprising - you don’t see any errors generated during execution and only see one final error(instead of 5) at the end.

A Real-life scenario

Now, let's explore how this behaviour can affect real-life batch job execution in D365FO.

Batch job stuck in Executing status

This is a performance problem. Imagine you have a critical bath job that should be completed within 1 hour. For example, you do some order reservations at 6 am and at 8 am your warehouse starts processing these reserved orders. You did all the performance testing (and verified that it always takes 1 hour) and have 1 hour between 7 am and 8 am as a buffer time. But someday, you notice that the job is still executing at 7 am and continues to execute at 8 am, which is when the problems begin. It keeps executing at 9 am, involving management in the troubleshooting process.

The root cause of this issue could be a simple error, such as a new unit of measure without a conversion setup. When the batch job encounters this error, the batch framework retries the job five times. The confusing part is that you don't see any errors in the log. Even if you ask Microsoft to check the top queries or server load, they won't find any unusual values.

To troubleshoot this issue, check the number of retries. If the number exceeds one, cancel the current execution, run it without batch mode, and investigate the error.

Email send procedure

Another scenario involves a batch job that sends reports as emails to customers. This job runs daily to execute the report. However, due to a problem with one email, some customers might receive five copies on a certain day while others receive none.

Fixing this behaviour may be complex. In one of our projects, we implemented an email send log to track what was sent and what was not, providing better visibility and control.

The opposite case - process waves procedure

Some developers are aware of this behaviour and try to avoid it (by using catch-all logic). For example, let’s review the process wave logic:

Some activities (e.g. running Release to warehouse from the user interface) automatically create multiple batch jobs to process waves.
Each batch job runs wave processing, which includes the picking step, where the system tries to find available on-hand inventory.
Since we have parallel jobs, they might select the same on-hand data, resulting in the error "Not available on-hand" for some jobs(which is a typical and expected situation).

The problem exists in WHSPostEngine::post(waveTable); method.

In this method, a try/catch block catches all errors and marks the batch job as "Processed" even if the processing failed.

In this case, a retry feature may be helpful, as the second run may find free items without affecting parallel jobs, but it is not used here.

Conclusion

The current implementation of D365FO batch jobs error handling is very confusing. The main problems are the default retry count of 5 (which would be more reasonable to set to 2 or 3) and that no log is saved for failed runs. Also, it would be great if the framework provided some default behaviour instead of developers having to write boilerplate code to set up retry logic.

I hope the examples presented in this blog post have helped you better understand the batch job process and its potential challenges. If you have any questions or want to share your D365FO batch job cases, please feel free to contact me.

Improve the speed for BACPAC database restore from Tier2 to Tier1

Wed, 01 Feb 2023 22:12:03 GMT

Accurate data on D365FO DEV VMs can be vital for the development process, especially during the Go-Live and Support phases. One of the issues is that development VMs use SQL Server and UAT/PROD environments use SQL Azure as a database. So to get a copy of the data, you need to convert SQL Azure BACPAC file to a standard SQL server format(you can use debugger on Tier2, but it may be not convenient in some situations).

SQL BACPAC is a zip file that contains table data in text format. If you change the .backpac extension to .zip and open it in the archive tool, you will see something like that:

So the BACPAC restore procedure creates all tables in the database and populates the data in these tables.

To load the data, Microsoft released a SqlPackage tool. You can run it directly or use d365fo.tools. Both commands are shown below:

#USE d365fo.tools
$StartTime = get-date 
Import-D365Bacpac -BacpacFile "J:\LCS\PreProd2022_05_24.bacpac" -ImportModeTier1 -NewDatabaseName PreProd2022_05_24_2
$RunTime = New-TimeSpan -Start $StartTime -End (get-date) 
WRITE-HOST "Execution time was $($RunTime.Hours) hours, $($RunTime.Minutes) minutes, $($RunTime.Seconds) seconds" 

#USE DIRECT SqlPackage CALL 
$StartTime = get-date 
cd C:\temp\
$fileExe = "C:\Temp\d365fo.tools\SqlPackage\SqlPackage.exe"
& $fileExe /a:import /sf:J:\MSSQL_BACKUP\pope_2022_05_16_small.bacpac /tsn:localhost /tdn:pope_2022_05_16_small_sp_ind  /p:RebuildIndexesOfflineForDataPhase=True /MaxParallelism:32 /p:DisableIndexesForDataPhase=FALSE
$RunTime = New-TimeSpan -Start $StartTime -End (get-date) 
WRITE-HOST "Execution time was $($RunTime.Hours) hours, $($RunTime.Minutes) minutes, $($RunTime.Seconds) seconds"

Sometimes this procedure may be slow, so let's discuss what parameters may be used to improve BACPAC restore time:

Get a fast hardware

Azure VMs are quite limited in terms of disk performance. A standard configuration may have SSDs or HDDs disks, but their speed will be much slower compared to local PCEe drives(e.g. Samsung 870 PRO).

The process is the following: you restore the BACPAC on the local machine, after the restore, create a backup of the SQL database, move it to the Azure file share and copy-restore this SQL backup on the cloud Azure VM. Working with native SQL backup is quite a fast operation compared to BACPAC conversion, and Local PC may provide a great advantage in restoring time. The sample code for this is the following:

$StartTime = get-date 
Import-D365Bacpac -BacpacFile "C:\Temp\PreProd2022_05_24.bacpac" -ImportModeTier1 -NewDatabaseName PreProd2022_05_24_2  -DatabaseServer "SDS-WS83" -SqlUser denis -SqlPwd "Pass" 
$RunTime = New-TimeSpan -Start $StartTime -End (get-date) 
WRITE-HOST "Execution time was $($RunTime.Hours) hours, $($RunTime.Minutes) minutes, $($RunTime.Seconds) seconds"

Use the Delayed durability database option

Delayed durability is a database option that speeds up certain database operations, but the database may stay in corrupted mode if the server crashes. D365FO database contains a huge number of objects, and the time for DB structure creation may be improved by using the Delayed durability flag.

The main problem is that it can be enabled only after the database is created, but SQLPackage doesn't have the option to automate this. So currently, I use this approach only with manual steps. In ~3-5 min after executing SQLPackage.exe, it creates a database, and then I manually run the following command for the newly created database.

USE [master]
GO
ALTER DATABASE [PreProd2022_05_24_2] SET RECOVERY SIMPLE WITH NO_WAIT
GO
ALTER DATABASE [PreProd2022_05_24_2] SET DELAYED_DURABILITY = FORCED WITH NO_WAIT
GO

Use RebuildIndexesOfflineForDataPhase switch

RebuildIndexesOfflineForDataPhase=True is one of the SQLPackage performance flags that rebuilds indexes with offline mode after the data load procedure. On my tests, I noticed some improvements with this flag, and it is something to keep in mind for a larger database.

Use DisableIndexesForDataPhase switch

By default SQLPackage disables table indexes before loading the data and then enables them after the load. For large tables, it should improve the data load performance, but for small tables, you get two additional steps: disable and enable indexes. /p:DisableIndexesForDataPhase=FALSE flag changes this behavior, it loads the data without recreating indexes. For my test files, it provided a significant performance boost.

Test results

In tests, I used the following BACPAC files:

A small file: 94MB BACPAC size, 2.2 GB SQL Server database size
A standard file: 2.5GB BACPAC size, 24.7 GB SQL Server database size

I compared two different environments:

A Local PC with a fast CPU and PCE SSD Intel core i9 10900 2.8GHz(up 4.8Gz)
A Standard development Azure VM D8v3 E5-2673 v4 (2.3Ghz) and 15HDDs

Test type	Small file/Azure	Standard file/Azure	Small file/Local	Standard file/Local
Default SQLPackage parameters	1 hour, 16:24	3 hours, 51:36	21:33	1 hour, 23:50
Disabled durability	46:35	3 hours, 18:36	20:11	1 hour, 20:12
Disabled durability, /p:RebuildIndexesOfflineForDataPhase=True		2 hours, 52:16		1 hour, 07:09
/p:DisableIndexesForDataPhase=FALSE		2 hours, 23:19	21:01	51:15

For the small file, the restore times varied from one hour to 20 minutes. What is interesting here is that for fast hardware, the restore time is almost independent of various switches. Delayed durability provided a significant boost for slow disk systems(standard DEV VM). Also, for a small amount of data, it is worth considering/p:DisableIndexesForDataPhase=FALSE flag

update 06/03/2023: you can auto clean log tables from the file using the following script

Clear-D365TableDataFromBacpac -Path "X:\MSSQL_BACKUP\AxDB.bacpac" -TableName "SECURITYOBJECTHISTORY","*Staging*","BatchHistory","SYSDATABASELOG*","ReqCalcTaskTrace" -ClearFromSource

Summary

I described different options for the BACPAC restore process. Of course, every database size is unique, but this should be a good place to start if you want to optimise the restore time for your project. For example, in my test, a "standard" size file may be restored in 4 hours or just in 1 hour, depending on different configurations.

What is also important is to have a process that covers the needs of your project. For example, you may have a standard automated restore process that takes 4h and have an "emergency" process that requires some interactions(e.g. Delayed durability, manually deleting large log tables from BACPAC before the restore, etc...) that can be done in 1h.

I hope you find this information useful. As always, if you see any improvements, suggestions or have some questions about this work, don't hesitate to contact me.

Understanding Dynamics 365FO different Tiers performance

Thu, 03 Nov 2022 22:12:03 GMT

ERP system performance is always an essential topic in system implementation. And with the cloud systems, it is even more complex as we have different system tiers. General planning is described in the following docs article: Environment Planning and we have the following environments:

But there is not much publicly available information related to D365FO hardware configuration for these Tiers. In this blog post, I describe some cases that help to understand Dynamics 365FO performance basics.

Different tiers and their performance

D365FO, as a backend, uses different database types: SQL Sever for Tier1 and SQL Server Azure for Tier2+ and PROD.

In the standard system, there is a Run performance test form that allows seeing the execution speed for different SQL queries. This is a single-threaded test that measures the speed of individual queries and not the overall capacity of the system.

I used 10k rows instead of the default 1k, ran it on several clients/environments and got the following results (to present the data in a table view, copy them to Excel and run Data > Text to Columns with ":" delimiter)

I also ran the test in Tier3 and Tier4, and it seems they are similar to Tier2.

Let's discuss these results:

Single-row operations

In operations that include many small calls SQL Azure is slower(4-8 times) than a standard SQL Server. This is probably happening due to a large latency. The difference looks dramatic(as a typical X++ code often contains many small select statements), but in the latest releases, Microsoft implemented a lot of caching classes, so for business operations, it is not so critical in general.

One of the real-life examples where you can see this difference is a display method with multiple SQL statements that is used in a form grid. Even with the caching, it may dramatically slow down the form behaviour. For example, 1 second overhead that is not even noticeable on the development VM may become 8 seconds on PROD and create "system is very slow" effects for users.

Multiple rows operations

There is not much difference between environments in bulk operations(like update_recordset). They are also faster compared to individual operations (e.g. row-by-row updates). You can often see advice to use these bulk operations by default, but I recommend avoiding this. The main issue is that update_recordset and delete_recordset are blocking operations and may stop the whole system if used in the wrong way. And for the cloud system, it will be very hard to troubleshoot and fix that.

I wrote an article regarding this Blocking in D365FO and why you shouldn't always follow MS recommendations, and my advice is to use them only if you can measure the business operation benefit.

Also, the fun fact is for the small/mid clients, your laptop probably is more performant than a PROD instance of a multi-million ERP system. Keep this in mind while discussing performance issues with Microsoft support

Environment planning

Let's discuss some considerations used for environment planning.

Pricing

The actual price depends on your agreements with Microsoft, but some indicative values are the following: A typical Tier1 is ~500$/month. Tier2 is 1400$/month, Tier3 is 4000$/month and Tier5 ~12000$/month.

But what is important is a billing period. For Tier1, it is per hour(you can stop the environment when it is not needed), but higher tiers are billed per month.

Microsoft is also preparing a new "Online development" environment that looks like a simplified Tier2, but its price is unknown now.

Tier1 or Tier2+ for large data load processes

If we check the performance, in most cases, Tier1 will be faster than Tier2. For one customer, we measured data load time for Released products/Product/Customers entities that ran in multiple threads. The time difference between Tier1(D8v3) and Tier2 was 200-250% (e.g. load run for 1h on Tier1 and 2-2.5h on Tier2)

Another point to consider is stability. Tier2+ databases are located on Azure SQL Database elastic pools, you can even execute the following command and see who your neighbours are:

SELECT * from sys.dm_user_db_resource_governance

It allows to scale dynamically, but when the Database is migrated to a different instance SQL TransientSqlConnectionError will be generated.

Another point is ongoing maintenance. A table may be reindexed by DAMS service during the data load Running 1M databases on Azure SQL for a large SaaS provider: Microsoft Dynamics 365 and Power Platform. This leads to "Could not complete cursor operation because the table schema changed after the cursor was declared" X++ error.

Some D365FO functions can’t handle such errors properly(on one project with 10.0.20, DAMS actions stopped multithreaded DIXF job after 3 hours of execution).

PROD performance

Microsoft may dynamically scale the PROD database level, depending on customer workload. The exact scale-up/down criteria are unknown, but I saw some cases where the PROD database was allocated even to a lower database tier than a standard Tier2 environment. So you need to be able to measure and compare the performance for critical operations to communicate with support properly.

Performance testing

You will see this advice in every Microsoft presentation: if you need to measure performance, always use Tier2+ instances. Also, performance testing should not be considered as a one-time exercise, the test should define some measures and a way to monitor them.

The overall flow for D365FO performance testing works this way:

You calculate the planned workload.
You submit it using the LCS Estimator tool.
Before GoLive, you need to fill out a GoLive readiness checklist. One of the questions will be whether you tested your system on exactly the same number of lines you specified in the LCS Estimator and whether you are happy with the results(on some projects they even required an official paper). You also need to provide dates of the testing and environment ID
Microsoft saves the monitoring data from this Environment during the specified dates

So after GoLive, they have this data for any arguments that the PROD is “not fast enough”. You also need this data to register a case if you have general performance issues

Applications servers

Now application servers are similar to 4-core CPUs and 16GB of RAM VMs. ~~You can't get better AOSes, but~~ Microsoft may add more servers if needed. Update 07/11/2022 - Several people reported that for the large number of users, AOSes may contain up to 50GB of RAM, so it seems it is possible to have a different config.

LCS provides a view for AOS CPU load and AOSes free memory, so it is quite easy to check if you have reached your capacity

One of the CPU-intensive operations on AOS is a Data management import. For every record, it scans all record fields with all extensions in all modules, which produces a significant workload on application server CPUs.

Summary

I described some performance measures related to different Dynamics 365 F&O environments. I hope you find this information useful for project planning. As usual, don't hesitate to contact me in case of any questions or comments.

Also, I can't find a working Tier5 environment. If you have Tier5 available, it would be great if you could share the "Performance form" results on this environment(with 10k records and when the system is not loaded).Update 04/11/2022 - it seems from a DB level Tier5 has the same speed as Tier2

D365FO upgrade. Transfer attachments from AX2012

Wed, 21 Sep 2022 22:12:03 GMT

An attachment migration procedure is one of the tricky parts you may face during the AX2012 - D365FO project upgrade. In this post, I describe an Attachment export/import journal solution for doing such migration.

D365FO Attachments types

As D365FO is a cloud-based system, attachment storage types may be different from what users get used to previously. So, let's discuss D365FO attachments types and the pros/cons of each of them:

Database: attachments are stored in the database in a blob field. This is perfect for text attachments, like notes. You can edit the attachment content, but only from the D365FO user interface. It increases the D365FO database size.
Azure storage: attachments are stored inside an internal Azure storage, managed by Microsoft. Users can't edit attachments directly(only download and upload files) as this storage account is accessible only from the D365FO user interface. This approach does not consume Database space, Microsoft charges for the storage separately.
SharePoint: attachments are stored on the SharePoint site, managed by clients. Users can edit attachments, and documents are available outside of the D365FO interface(for example, you may use a group in Teams and link a SharePoint folder to it). But in this case, the customer needs to maintain the SharePoint site.

Transfer attachment module description

There are already several ways of transferring attachments while doing a migration project from AX2012 to D365FO, but in this post, I will describe the additional method, and probably it can be called "yet another way to transfer attachments from AX2012".

Some of the out-of-the-box ways for attachment migration are:

MoveDocumentsToDatabase job from the FastTrack team. It moves all attachments to the AX2012 database that can be used in the upgrade process.
Standard entities for attachments creation(e.g. CustomerAttachmentsV2Entity). In the case of entity usage, you need to create a ZIP data package that contains all files and entity data.

The reason for the module development is to provide more flexibility and control over AX2012 attachment transfer. Let's check how it works.

The module consists of AX2012 and D365FO parts.

AX2012 functionality

It is an XPO project that needs to be imported into an AX2012 live application. After the import, you need to run the DEVDocuExport menu item.

It will display the following window:

This dialog allows you to specify a table and set up some filters for it(e.g. export attachments only for not blocked customers).

As a result of this run, the system exports all attachments for the selected table to the directory and creates a Descriptor file that links the primary table key to an attachment file.

The Descriptor file gives you great flexibility, e.g. you can rename table keys or change document types mapping.

The next step is to copy the created directory with AX2012 attachments to Azure storage. It can be done by using the following power shell command.

cmd.exe /C "cmdkey /add:`"aaavmstorage.file.core.windows.net`" /user:`"localhost\aaavmstorage`" /pass:`"BQubgv1NlZE+LtbbL8jKLrkVSwBlLHGuygbu9fy6gDv+UiRTiw63cpmVlAEVhSLCup1XRbbVajt6ww==`""
#Mount the drive
New-PSDrive -Name Z -PSProvider FileSystem -Root \\aaavmstorage.file.core.windows.net\aaavmfileshare

Copy-Item "C:\Attachments" -Destination "Z:\Attachments" -Recurse

D365FO part

At that stage, we have our attachments in the cloud, and it is time to import them into D365FO.

After installing the required X++ objects, open the following link

https://mycompany.operations.dynamics.com/?cmp=CMP&mi=DEVDocuExpImpJournalTable

It will open the Import attachment form.

The first step is to go to the Journal name field(View detail) and create a new Journal type. A Journal type defines a reference to Azure file share (a connection string and a share reference).

Then you need to specify the Azure location(folder with files) for our import and press the Test connection button to validate the settings.

After Journal creation, the next step is to import lines from the Excel Descriptor file generated previously during the AX2012 export(Import lines from Excel button).

It will create journal lines(one line per file).

The final step is to load attachments to D365FO, you can do this only for one line(to test the process) or for the whole journal. It will use the standard "Attach document" procedure, which creates attachments according to the document type settings.

And as a result, you get a destination table record with attachments(e.g. attachments for the customer)

Summary

The described solution provides excellent flexibility for moving attachments from AX2012 to D365FO. You can find all code used in this post in the following folder(AX2012 project saved as a resource): https://github.com/TrudAX/XppTools/tree/master/DEVTutorial/DEVDocuExpImp.

I hope you find this information useful. As always, if you see any improvements, suggestions or have some questions about this work, don't hesitate to contact me.

XppInteg - Import purchase orders from XML files into D365FO

Wed, 04 May 2022 22:12:04 GMT

This post extends my several previous posts but with another messages format. We use an XML format and try to import files in X++ from Azure File Share and create purchase orders.

I use a very simplified example of integration with the INFOR system. This system has an integration component that can export the data in XML format, and we will process(import) these files in D365FO.

Reading XML data

What is good that a file generated from INFOR is probably the most complex format that you may see in XML integrations. It has a custom namespace, data in tags, data in attributes, data in the similar elements. Here is a sample file context:

<?xml version="1.0" encoding="UTF-8"?>
<SyncPurchaseOrder xmlns="http://schema.infor.com/InforOAGIS/2" languageCode="en-US">
  <DataArea>
    <PurchaseOrder>
      <PurchaseOrderHeader>
        <DocumentID>PO00001</DocumentID>
        <DocumentDateTime>2022-04-16</DocumentDateTime>
        <SupplierParty>
          <PartyIDs>
            <ID accountingEntity="USMF">1001</ID>
          </PartyIDs>
        </SupplierParty>
        <ShipToParty>
          <Location type="Warehouse">
            <ID accountingEntity="USMF">11</ID>
          </Location>
        </ShipToParty>
        <UserArea>
          <Property>
            <NameValue name="eam.UDFCHAR01" type="StringType">Requestor Name</NameValue>
          </Property>
        </UserArea>
      </PurchaseOrderHeader>
      <PurchaseOrderLine>
        <LineNumber>10</LineNumber>
        <Quantity unitCode="EA">15</Quantity>
        <UnitPrice>
          <Amount currencyID="AUD">10</Amount>
        </UnitPrice>
        <UserArea>
          <Property>
            <NameValue name="eam.UDFCHAR01" type="StringType">M0001</NameValue>
          </Property>
        </UserArea>
      </PurchaseOrderLine>
      <PurchaseOrderLine>
        <LineNumber>20</LineNumber>
        <Quantity unitCode="EA">2</Quantity>
        <UnitPrice>
          <Amount currencyID="AUD">10</Amount>
        </UnitPrice>
        <UserArea>
          <Property>
            <NameValue name="eam.UDFCHAR01" type="StringType">M0004</NameValue>
          </Property>
        </UserArea>
      </PurchaseOrderLine>
    </PurchaseOrder>
  </DataArea>
</SyncPurchaseOrder>

We will read this file into the two staging tables(header and lines) and then create a purchase order based on these tables.

In order to deal with such a complex format I created a helper class DEVIntegXMLReadHelper. It uses the standard System.Xml* methods and provides several helper methods that simplify the code structure:

//set up namespace alias(used in search)
readHelper.initNamespace(xmlDoc, 'infor', @'http://schema.infor.com/InforOAGIS/2');

//read a typed data from the specified tag
header.PurchId  = readHelper.xmlGetValueNodeStr(nodePOHeader, 'DocumentID');
line.PurchPrice = readHelper.xmlGetValueNodeReal(nodePOLine, 'UnitPrice/Amount');

//search node inside the specified node by path
node = readHelper.xmlGetNodeByPath(nodePOHeader, 'SupplierParty/PartyIDs/ID');
header.OrderAccount = node.get_InnerText();

//search node inside the specified node by search string
node = readHelper.xmlSelectSingleNode(nodePOHeader, 'infor:UserArea//infor:Property//infor:NameValue[@name=\'eam.UDFCHAR01\']');
header.RequestorName = node.get_InnerText();

//getting the data from the attribute
xmlAttributeCollection = node.get_Attributes();
xmlAttribute = xmlAttributeCollection.get_ItemOf("accountingEntity");
header.CompanyId    = xmlAttribute.get_InnerText();

The full class is here DEVIntegTutorialProcessPurchConfirmXML

Solution description

In this blog post, I describe a solution(with "Consuming external web services" type) for a D365FO Custom Service that will talk to the Integration Platform. It will do the following:

Process purchase orders for different companies.
Avoid issues like OData or DMF like throttling, monitoring failed batch imports and rollback of partial failures.
Read a custom data format in XML format from Azure File share as messages (sourced from an external program).
Provide a solution to log and monitor the failures easily.
Perform custom transformations within D365FO as part of the import.

Our integration should read messages from the Azure file share and create purchase orders in D365FO.

The diagram that describes the process is below:

Proposed solution

In the following section, I provide a solution that can be used as a starting point to import and process messages from Azure file share.

Initially, our XML files will be loaded to the Azure storage and then processed by D365FO.

Connection types form

Connection types form allows specifying the Connection type resource(currently, it supports Azure file share, Azure Service Bus and SFTP) and connection details for the selected resource.

For storing a connection string, there can be three options:

Enter the value in this form as plain text
Use encrypted value
Store the connection string value as a Secret in the Azure Key Vault. In this case, the user needs to provide a reference to the standard Key Vault name (steps to set up it are described in the following post: Azure Key Vault & MSDyn365FO: Setup Certificates and Passwords).

Inbound message types form

The next form to describe our integration will be the Inbound message types form:

This form contains three main sections:

1 - Details tab

Defines Azure file share connection and a path to folders to import messages.
Contains a link to the custom Class that will do processing from the queue. The class should extend the base class DEVIntegProcessMessageBase and implement the following method:

abstract void  processMessage(DEVIntegMessageTable  _messageTable, DEVIntegMessageProcessResult _messageProcessResult)
{
...
}

The integration engine will call this method outside of a transaction so that all transaction control can be implemented per Message type. There can be different options here: one transaction per message, multiple transactions(for example, if a file contains several independent journals) or a single transaction per line. The result of processing and the current stage should be written to _messageProcessResult variable, so in case of an unhandled exception, this information can be saved for review. Also, this class will be created one time per entire import session so it can implement different caching options.

2 - Operation parameters tab

Contains parameters that are individual for the current operation. In our case, we don't have any parameters.

3 - Advanced settings tab

Contains some common parameters:

The number of parallel threads to process incoming messages.
Reading limits from the queue(e.g. to read only the first three messages, this is for testing purposes)

Also, this form contains service operations:

Test connection button that tries to connect to the specified Azure file share directory and read files from it.
View messages button that display messages with this type
Import file button that can be used in testing scenarios to manually import a message from a user's computer without connecting to Azure file share.

Incoming messages form

This table will store the details of each incoming message.

Every message has a status field that can contain the following values:

Ready – a message was read to D365FO
Cancel – The user has decided not to process the message. This is an alternative to a delete option
In process – system-generated status, a message is processing now.
Error – failed validation or errors during the processing
Processed – completed successfully

In this form, it is also possible to do the following operations:

View incoming message context
Filter by different statuses
View a detailed error message
Change the status to process the message again
View file processing statistics (processing duration, time, number of lines)
View the processing attempt counter. It allows implementing scenarios like trying to process messages with "Error" status several times.

Load incoming messages operation

It is a periodic batch job that we can run for one or multiple message types.

It connects to the Azure file storage, reads a file, creates a record in the Incoming messages table with Ready status and attaches a message content to this record. After this, the file is moved to the Archive folder. If Run processing is selected, the system will execute the processing of the loaded messages.

Process incoming messages

Message processing may be executed as a separate operation - Process incoming messages that selects all not processed messages and calls the processing class for them.

The logic of how to process the file is different per message type/class. For a simple scenario, the class can just read the message content and create some data in one transaction. For this blog post, I implemented two-step processing. Also, for the Process operation, you can specify the number of Processing attempts per file. The system tries to process a message and in case of any error, it increases the message Processing attempts counter. This allows retrying processing several times without a user involvement.

See a sample diagram below:

The class reads the message data and writes data into a staging table during the first step. Then based on the staging data values, a new purchase order is created. It can be done using data entities or just directly writing to tables.

Error types and how to handle them

It is not a big task to create a purchase order based on some data. The complexity of integration is often related to exception processing and error monitoring. Let's discuss typical errors and how users can deal with them.

Connection errors

A batch job exception will be generated if our batch job can't connect to an Azure file share or read files. It is a configuration error requiring a system administrator's attention. Notification may be done using a batch job status. After troubleshooting, the system administrator can use the "Test connection" button to validate that the system can now read files from Azure file share.

File format errors

The next error type is the wrong file format, so we can't even read the data from a message.

To test this case, I renamed one XML element. After the import, users will see this message with the Error status. Notification may be done using standard filtering by the Status column.

Users can view an error log, then download the message and check the reason for this error. For example, there may be the following reasons:

Our code that processes messages contains some mistakes. In this case, we can download and send this message to a developer to fix the logic. The main advantage of this solution is that users can run processing without connecting to Azure by using the manual Import message button. After fixing the problem, we can rerun the Processing.
External system sent a message in the wrong format. In this case, the user can send this file back to the external party and then change the message status to Cancel.

Data errors

The message has a correct structure but contains incorrect data(e.g. values that don't exist). In this case, a Status of our Message will be Error, and an error log will be generated.

Users can view this error, display a Staging data to check the values from the message and take some actions(e.g. create missing values in the related tables if they are valid). After that, they can Process this message again.

In some implementations(EDI), we can even allow staging data editing. A similar type of error is a posting error.

Wrong result errors

That is probably the worst scenario. The message was processed successfully, but the resulting document contains some incorrect data.

Users can view the staging data and check that they are correct to analyse the result. Also, a message content can be sent to a developer, and they may trace the document processing using the manual Import message button

Summary

I provided a sample implementation for an Azure File Share integration for D365FO. The main concept of it is to create a basic framework to simplify troubleshooting(most typical errors and all related data can be viewed in one form - Incoming messages) and provide some additional logging.

In the series of posts I provided samples for different file formats used for import:

XML
CSV
JSON

Different most complex D365FO documents:

Sales order
Purchase order
Ledger journal
Standard DMF project

And different exchange media:

Azure File Share
Azure Service Bus
SFTP

This may or may not be appropriate in your case(there are different options how to implement this). Anyway, I recommend using the following checklist while designing the integration: Integration solution specification.

Files used for this post can be found in the following folder

I hope you find this information useful. As always, if you see any improvements, suggestions or have some questions about this work, don't hesitate to contact me.

Dynamics AX performance monitoring: missing indexes

Fri, 28 Jan 2022 22:12:03 GMT

Proper SQL indexes play a vital role in the performance of Dynamics AX(or D365FO on-premise). Even one query that doesn't have appropriate indexes may slow down the system and increase the load on the SQL Server. In this post, I want to describe how to implement a monitoring solution for missing indexes.

Missing indexes example

Let's simulate the missing indexes issue. I am using a standard AX2012 Demo VM for this. For example, we need to find all vendor invoice lines related to a sales order.

To do this, a developer may write the following code:

    VendInvoiceTrans        vendInvoiceTrans;
    ;
    while select vendInvoiceTrans
        where vendInvoiceTrans.InventRefTransId == "002862"
    ...

The problem is that InventRefTransId field is not indexed, so to get the result, SQL Server needs to scan all rows in VendInvoiceTrans table. Depending on where this code is used, it may affect user performance or increase the load to a SQL Server. Also, InventRefTransId is a highly selective field, so it is a perfect candidate for an index.

SQL Server query optimiser can automatically detect such cases and generate recommendations for index creation. These recommendations can be displayed by querying missing index view. Moreover, analysing the output of this view is a prominent part of any performance audit.

From some real-life experience of Dynamics AX performance optimisation, usually, it is enough to analyse 50 first rows from this view with an impact of more than 99%. Most(but not all) of such recommendations will be valid, and new indexes should be created.

But you should not automatically apply every recommendation. Here are some exceptions where you don't need to create an index even if the impact is 99%:

A "select" is an external query with a missing DATAAREAID/PARTITION field and a field already presented in the index: In this case, it is better to add DATAAREAID/PARTITION to such an external query.
There may be a lot of fields in the missing index recommendation, but not all are selective. Extensive indexes may slow down the system, so the number of fields for the new index must be minimal, you should add only selective fields to the new index.
A field is already presented in the index but not in the first place. A perfect example of this is CustPackingSlipJour table. To get a packing slip for a sales order, a Module(RefNum) should be specified. Usually, it contains just one value(Sales order type), and developers often forget to select it.

Another variation of this case is my example with VendInvoiceTrans above. In this case a code contains a condition for InventRefTransId field but the value for InventRefType is not specified. This is a development error, and both fields should be specified in the code and in the index.

Let's discuss how we can automatically monitor such issues.

Missing indexes monitoring solution implementation

Such problems are pretty easy to fix; you need to periodically check the missing indexes view. To automatically perform this task, I created the following Dynamics AX performance missing indexes monitoring procedure:

msdb.dbo.[AXMissingIndexesMonitor] @DBName = 'MicrosoftDynamicsAX',
@SendEmailOperator = 'axoperator', @DisplayOnlyNewRecommendation = 0, @Debug = 0

The idea is the following:

You create a SQL Agent job to run this procedure every day. If there are any missing indexes with 99% impact in the first 50 rows of the missing index view, the system sends an email to the person responsible for Dynamics AX performance monitoring.

The developer should always react to this email. There can be two outcomes:

Either the index should be created(it should be done in AX application level and new application release deployed)
Or if missing index recommendation is not valid a developer may mark it approved(set the flag IsApproved from AXTopMissingIndexesLog table)

D365FO Cloud version

The manual monitoring is valid only for On-premise D365FO versions(based on SQL Server). The Cloud version of Dynamics 365 for Operations uses Azure SQL Database at the backend, and missing indexes are created automatically by the DAMS service. The following article is an excellent description of how the process works internallyRunning 1M databases on Azure SQL for a large SaaS provider: Microsoft Dynamics 365 and Power Platform.

Conclusion

Using the described solution, you can monitor your Dynamics AX SQL indexes and get a notification when SQL Server identifies new recommendations. It will help to maintain a stable level of Dynamics AX performance.

The code for this can be found in the following folder. This procedure is a valuable addition to my previous monitoring solutionMonitoring the most problematic performance problem in Dynamics AX - parameters sniffing.

I hope you find this information useful. Don't hesitate to contact me in case of any questions or if you want to share your Dynamics AX/D365FO SQL monitoring approach.

XppInteg - Azure Service Bus integration solution for Dynamics 365 FinOps

Thu, 11 Nov 2021 22:12:04 GMT

This post extends my previous post How to implement file-based integration in Dynamics 365 FinOps using X++ but with another way to transfer messages. As the file-based approach still may work for many cases, in the cloud world, we have a lot of other, fancier ways to transfer messages. In this post, I will describe how to use Azure Service Bus to do a custom X++ based integration.

Azure Service Bus is a cloud-based messaging service providing queues with publish/subscribe semantics and rich features. It is a fully managed service with no servers to manage or licenses to buy.

Common Service Bus usage scenarios:

Build reliable and elastic cloud apps with messaging.
Protect your application from temporary peaks.
Distribute messages to multiple independent backend systems.
Decouple your applications from each other.

The pricing for Service Bus starts from 10USD per month for 12M operations.

Setting up Azure service bus

To set up a new Service Bus, we need to create a new Service Bus resource in the Azure portal and set up a new Queue for incoming messages.

Then go to the Shared access policies menu and create a new access key for the Service Bus. Copy Primary Connection String for this key.

Then you may install and run ServiceBusExplorer, a handy utility to manage Service Bus from a local computer https://github.com/paolosalvatori/ServiceBusExplorer/releases

Solution description

In this blog post, I try to describe a solution(with "Consuming external web services" type) for a D365FO Custom Service that will talk to the Integration Platform. It will do the following:

Process sales orders for different companies.
Avoid issues that OData has with bulk sales orders import like throttling.
Avoid issues that DMF has with monitoring of failed batch imports and rollback of partial failures.
Read a custom data format in the Azure Service Bus queue as messages (sourced from an external program).
Provide a solution to log and monitor the failures easily.
Perform custom transformations within D365FO as part of the import.

The first step for building such integration is to create a mapping document. I put a sample template here - Field Mapping sample. For this post, the format for Sales order will be a simple custom JSON document that describes the sales order header and lines.

{    "CompanyId": "USMF",
    "customerId": "US-002",
    "externalOrderNumber": "ABCDEFG",
    "lines": [
        {
            "lineQuantity": 1,
            "linePrice": 11,
            "lineItemBarcode": "M0001"
        },
        {
            "lineQuantity": 2,
            "linePrice": 22,
            "lineItemBarcode": "M0004"
        },
        {
            "lineQuantity": 3,
            "linePrice": 33,
            "lineItemBarcode": "M0007"
        }
    ] }

Our integration should read messages from the Service Bus queue and create sales orders in D365FO.

The diagram that describes the process is below:

Proposed solution

In the following section, I provide a solution that can be used as a starting point to import and to process messages from Azure Service Bus.

Connection types form

Connection types form allows specifying the Connection type resource(currently two resources are supported: Azure file share and Azure Service Bus) and connection details for the selected resource.

For storing connection string, there can be two options:

Enter the value in this form
Store the connection string value as a Secret in the Azure Key Vault. In this case, the user needs to provide a reference to the standard Key Vault name (steps to set up it are described in the following post: Azure Key Vault & MSDyn365FO: Setup Certificates and Passwords).

Inbound message types form

The next form to describe our integration will be the Inbound message types form

This form contains 3 main sections:

1 - Details tab

Defines Service Bus connection and a Queue name from which to import messages.
Contains a link to the custom Class that will do processing from the queue. The class should extend the base class DEVIntegProcessMessageBase and implement the following method:

abstract void  processMessage(DEVIntegMessageTable  _messageTable, DEVIntegMessageProcessResult _messageProcessResult)
{
...
}

This method will be called by the integration engine outside of a transaction, so all transaction control can be implemented per Message type. There can be different options here: one transaction per message, multiple transactions(for example, if a file contains several independent journals) or a single transaction per line. The result of processing and the current stage should be written to _messageProcessResult variable, so in case of an unhandled exception, this information can be saved for reviewing. Also, this class will be created one time per entire import session, so it can implement different caching options.

2 - Operation parameters tab

Contains parameters that are individual for the current operation. In our case, we don't have any parameters.

3 - Advanced settings tab

Contains some common parameters:

The number of parallel threads to process incoming messages.
Reading limits from the queue(e.g. to read only the first three messages, this is for testing purposes)

Also, this form contains two service operations:

Check connection button that tries to connect to the specified queue and read(with Peak only) the first three messages.
Import message button that can be used in testing scenarios to manually import a message from a user computer without connecting to Service Bus.

Incoming messages form

This table will store the details of each incoming message.

Every message has a status field that can contain the following values:

Ready – a message was read to D365FO
Hold – The user has decided not to process the message. This is an alternative to a delete option
In process – system-generated status, a message is processing now.
Error – failed validation or errors during the processing
Processed – completed successfully

In this form, it is also possible to do the following operations:

View incoming message context
Filter by different statuses
View a detailed error message
Change the status to process the message again
View file processing statistics (processing duration, time, number of lines)
View the processing attempt counter. This allows implementing scenarios like trying to process messages with “Error” status several times.

Load incoming messages operation

It is a periodic batch job that we can run for one or multiple message types.

It connects to the Service Bus, reads messages(with Peek&Delete type), creates a record in the Incoming messages table with Ready status and attaches a message content to this record. If Run processing is selected, the load system will execute the processing of the loaded messages.

Process incoming messages

Message processing may be executed as a separate operation - Process incoming messages that selects all not processed messages and calls the processing class for them.

During the first step, the class reads the message data and writes data into a staging table. A sample code for this:

tutorialSOHeaderContract = FormJsonSerializer::deserializeObject(classNum(DEVIntegTutorialSOHeaderContract), serializedJSONstr);

Then based on the staging data values, a new sales order is created. It can be done using data entities or just directly writing to tables.

salesLine.initValue();
salesLine.ItemId = salesLineStaging.ItemId;
salesLine.initFromSalesTable(salesTable);
salesLine.initFromInventTable(InventTable::find(salesLine.ItemId));
salesLine.SalesQty              = salesLineStaging.SalesQty;
salesLine.QtyOrdered            = SalesLine.calcQtyOrdered();
salesLine.SalesDeliverNow       = salesLine.SalesQty;
salesLine.setInventDeliverNow();
...
salesLine.createLine(true,false,false,true);

Error types and how to handle them

It is not a big task to create a sales order based on some data. The complexity of integration is often related to exception processing and error monitoring. Let's discuss typical errors and how users can deal with them.

Connection errors

A batch job exception will be generated if our batch job can't connect to a Service Bus or read messages. It is a configuration error, and it requires a system administrator's attention. Notification may be done using a batch job status. After troubleshooting, the system administrator can use the "Test connection" button to validate that the system can now read messages from the Service Bus.

File format errors

The next error type is a wrong file format, so we can't even read the data from a message.

To test this case, I renamed several columns. After the import, users will see this message with the Error status. Notification may be done using standard filtering by the Status column.

Users can view an error log, then download the message and check the reason for this error. There may be two reasons:

Our code that processes messages contains some mistakes. In this case, we can download and send this message to a developer to fix the logic. The main advantage of this solution is that users can run processing without Service Bus by using the Import message button. After fixing the problem, we can run the Processing again.
External system sent a message in the wrong format. In this case, the user can send this file back to the external party, then change the message status to Hold.

Data errors

The message has a correct structure but contains incorrect data(e.g. values that don't exist). In this case, a Status of our Message will be Error, and an Error log will be generated.

In some implementations(EDI), we can even allow staging data editing. A similar type of error is a posting error.

Wrong result errors

That is probably the worst scenario. The message was processed successfully, but the resulting document contains some wrong data.

To analyse the result, users can view the staging data and check that they are correct. Also, a message content can be sent to a developer, and they may trace the document processing using the manual Import message button

DMF integration

The text above was for the custom-written integration, but the same engine may be used for importing data with the standard Data Management module. The idea is the same as for the file integration - Multicompany DMF integration in Dynamics 365 FinOps using X++, in the message parameters, we can specify a DMF processing class and the entity from the Data management import project.

This class connects to Service Bus, saves the message to a temporary file and executes Data management project import for this file. This allows you to use a full set of Data entities, settings and transformations that Data management can provide. Let's consider the simple scenario: Customer group import

First you need to create an import project based on the sample XML

<?xml version="1.0" encoding="utf-8"?>
<Document>
 <CUSTCUSTOMERGROUPENTITY>
  <CUSTOMERGROUPID>MyTestGroup</CUSTOMERGROUPID>
  <DESCRIPTION>MyTestGroup Description</DESCRIPTION>
 </CUSTCUSTOMERGROUPENTITY>
</Document>

Setup a new Inbound connection type(based on DMF class) and specify the link to the import project.

Then load a message to our Service Bus queue and run the import

As the result a new message will be created and after processing a new Customer group record will be created

You can even implement a more complex scenarios, for example, import sales orders by using Composite data entities - Dynamics 365 Operations | Composite data entity – Sales order import (Only works with XML). However, this is not something that is recommended by Microsoft: "Composite entities are not recommended" from 5 Tips to improve the performance in a DMF import project.

Summary

I provided a sample implementation for an Azure Service Bus integration for D365FO. The main concept of it is to create a basic framework to simplify troubleshooting(most typical errors and all related data can be viewed in one form - Incoming messages) and provide some additional logging.

Files used for this post can be found in the following folder

I hope you find this information useful. As always, if you see any improvements, suggestions or have some questions about this work, don't hesitate to contact me.

Working with LedgerDimension values from posted transactions in D365FO

Thu, 14 Oct 2021 22:12:03 GMT

Recently we faced and fixed an interesting bug. We had a custom procedure that reversed a posted ledger journal. The logic of this procedure is quite simple - take the existing values from LedgetDimension fields for posted lines and create the same transactions but with the negative amount.

So the simplified version of the code looked like this:

debitAccount  = ledgerJournalTransSource.LedgerDimension;
creditAccount  = ledgerJournalTransVariance.LedgerDimension;
//Do the posting using these account

Then suddenly it stopped working with the following error:

Ledger account with 511-323-300 with record ID 1111 does not match the current account and/or advanced rule structure associated with structure

After some troubleshooting, we found out that the customer recently had updated settings for Ledger account structures.

In X++ a LedgerDimension type is a reference to 3 entities:

Dimension structure
Ledger default account
Default dimension

So the error basically said that we couldn't use the same Dimension structure again.

The fix was quite simple - extract a Default account and Default dimension from the old LedgerDimension value and create a new LedgerDimension using these extracted values:


debitAccount  = LedgerDimensionFacade::serviceCreateLedgerDimension(              LedgerDefaultAccountHelper::getDefaultAccountFromLedgerDimension(DebitAccount),
LedgerDimensionFacade::getDefaultDimensionFromLedgerDimension(DebitAccount));

creditAccount  = LedgerDimensionFacade::serviceCreateLedgerDimension(
LedgerDefaultAccountHelper::getDefaultAccountFromLedgerDimension(CreditAccount),
LedgerDimensionFacade::getDefaultDimensionFromLedgerDimension(CreditAccount));

After this change, the posting was successful. It was quite a tricky fix, hard to see at first.

So, I am trying to maintain a helper class that contains references to common ledger dimension operations(like in this post), feel free to check it and comment if something is missing. https://github.com/TrudAX/XppTools/blob/master/DEVCommon/DEVCommon/AxClass/DEVDimensionHelper.xml

D365FO Infolog call stack

Fri, 24 Sep 2021 20:12:03 GMT

Infolog is a primary message display system in Dynamics 365 FO. And 'infolog.add()' is probably the most favourite place for developers to insert breakpoints. The reason for this is that the source of some messages may be unclear and you need to use the debugger to understand the reason for a particular message.

In this post, I describe an X++ utility that saves stack traces for every infolog message. It allows, in some cases, understanding the source of the message without debugging. The idea of this post is based on the following post on AxForum.

Functional description

In order to display the last infolog messages, a new form was created in System administration - Inquiries - DEV Tools - Call stack

The form has 2 tabs:

Data tab that displays saved infolog messages and X++ call stack for the current message

Setup tab that allows setting up a logging level per user. You can also setup logging to all users for Warning and Error messages

Technical description

The modification is executed by subscribing to Info.onInfoLogMessageAdd() delegate. Then Call stack is saved by using xSession::xppCallStack(); method. Below is the sample code for this

class DEVCSXppCallStackHandler
{
    [SubscribesTo(classStr(Info), delegateStr(Info, onInfoLogMessageAdd))]
    public static void Info_onInfoLogMessageAdd(InfoLogMessageAddEventArgs _eventArgs)
    {
        DEVCSXppCallStackTable       callStackTable;
        boolean                      isEnabled;
        DEVCSXppCallStackUserSetup   callStackUserSetup;
        callStackUserSetup = DEVCSXppCallStackUserSetup::findCurrent();
        if (callStackUserSetup.RecId && callStackUserSetup.IsEnabled)
        {
            isEnabled = true;
            switch (callStackUserSetup.SysInfologLevel)
            {
                case SysInfologLevel::None:
                    isEnabled = false;
                    break;
                case SysInfologLevel::Warning:
                    if (_eventArgs.exception== Exception::Info)
                    isEnabled = false;
                    break;
                case SysInfologLevel::Error:
                    if (_eventArgs.exception == Exception::Info ||
                        _eventArgs.exception == Exception::Warning)
                    isEnabled = false;
                    break;
            }
        }
        if (isEnabled)
        {
            UserConnection       userConnection = new UserConnection();
            callStackTable.setConnection(userConnection);
            callStackTable.clear();
            callStackTable.initValue();
            callStackTable.Exception = _eventArgs.exception;
            callStackTable.InfologStr = _eventArgs.message;
            callStackTable.CallStack = xSession::xp CallStack();
            callStackTable.insert();

Then form DEVCSXppCallStackTable displays the saved data.

An additional tool that may be combined with this one is Arbela AOT Browser that allows viewing the code on Tier1 environments without Visual Studio usage.

Trace dialog calls (update 05.11.2023)

The Trace dialog calls option allows you to see the class and the stack trace information of any dialog form in D365FO. You can enable or disable this option for each user by checking or unchecking the Trace dialog calls checkbox in the “Dev Tools – Call stack” form.

To demonstrate how this feature works, I ran some periodic operations that use dialogs, such as:

History by transaction report
Create prenotes
Rebate management

These operations use different types of dialogs, such as Reports, Runbase classes, and SysOperation classes.

After running the operations, I opened the Trace form and saw the class and the X++ stack trace information for each dialog

This feature is very handy when you need to quickly identify the related classes and operations in D365FO. I have used it several times during calls with business users and it works very nicely.

Summary

You can download the tool using the following link - https://github.com/TrudAX/XppTools#installation. I hope you may find this tool useful, also feel free to comment if you see something is missing.

Multicompany DMF integration in Dynamics 365 FinOps using X++

Wed, 11 Aug 2021 22:12:03 GMT

In one of my last posts I described a sample approach for a periodic file-based integration using X++ custom code. The external system creates files on the Azure file share, then D365FO pulls these files, parses and processes them. One of the comments on this post was why DMF is not used for such tasks.

That is a perfectly valid comment, given that if you want to develop a custom import you need to do a lot of programming, like create staging tables and forms, implement a file parsing and create the final document via X++ code.

In this blog post I will show how DMF can be used in the same framework in order to implement a periodic files import.

Task description

We have an incoming folder in the Azure file share that contains a set of files and we need to import them via DMF. In this example I will use a “Customer group” entity in Excel format. Also let's implement a multi-company import. A company code will be specified for a file as a first characters before "_".

We need to import these files in D365FO and view the import status per file.

Proposed solution

In the following section, I provide some code samples that can be used as a starting point to implement a periodic file import and processing.

This form is used to define a connection to a cloud file share. In this example we will use a Azure file share link.

Inbound message types form

In this form we need to create a new class that processes our files. The logic for this will be quite simple: load a file, change the company, pass the file to a DMF framework. The DMF call will be synchronous, so we can get the status from DMF after the processing and update our message table.

On this form we can also specify for which entity we need to run our import. Basically it links the Incoming directory with a DMF project/entity. The same processing class can be used for different entities.

Also, this form contains two servicing operations:

Check connection button that tries to connect to the specified directory
Import file button that can be used in testing scenarios to manually import a file from a user computer without connecting to a network

Incoming messages form

This table will store the details for each inbound file. It displays the original incoming file name and its status.

In this form it is also possible to do the following operations:

View incoming file context
Filter by different statuses
View a detailed error message
Change the status to process the message again
View file processing statistics (processing duration, time, number of lines)

Load and process incoming files

It is a periodic batch job that we can run for one or multiple message types.

It connects to the shared folder, reads files, creates a record in the Incoming messages table with Ready status, attaches a file content to this message and moves the file to an Archive directory. If Run processing is selected, after the load, the file will be processed.

Internally the processing takes a file and run the same function that the user may run for a manual import in the Data management module

The sample code for this is below. You can find the same samples in several places in the system

//read a file..
//change the company from the file name...
//find a linked DMF definition import and path the file into it..
    
DMFExecutionId executionId = DMFUtil::setupNewExecution(definitionGroupEntity.DefinitionGroup);
//...
    DMFStagingWriter::execute(
        executionId, //_executionId
        0, //_batchId
        true, //_runOnService
        false, //_calledFrom
        DateTimeUtil::getUserPreferredTimeZone(), //_timeZone
        false //_compareData
    );
//...
}

As the process is synchronous, we can check the status after the import and update the Message table with results.

Error types and how to handle them

Let's discuss typical errors and how users can deal with them. Also I will compare them with a custom made import.

If our batch job can't connect to a File share or read and move files, a batch job exception will be generated. It is a configuration error and it requires system administrator attention. Notification will be done using a batch job status. After troubleshooting the error system administrator can use the "Test connection" button to validate that the system can now connect to the file share.

File format errors

The next error type is a wrong file format, so we can't even read the file content.

To test this case I renamed one of the columns

This file caused a 1 minute delay for DMF and then an internal error

DMF uses a data integration component that connects to an Excel file and that is probably why the error is not verbose.

This message is stored in the Messages table, users can view the error log, then download the file and check the reason for this error.

Data errors

The file has a correct structure but contains a wrong data(e.g.. values that don't exist)

In this case, a Status of our Message will be Error and an Error log will be generated.

Users can view the error detail using the standard DMF Execute history form, that displays invalid lines

The main difference from the custom import is that you can't implement "all or nothing" principle as DMF doesn't support transactions, lines from this file that passed validation were processed successfully.

Posting documents

We don't have a posting code in this example, but it can be implemented by modifying the processing class after the successful file load.

We can see this architecture in a lot of projects, where DMF is used only to load the data in staging tables that don't contain any business logic and the chance of error is minimal.

Then in order to process these staging records(create business documents) a custom code is executed. In this case you rely on DMF only for reading the file and implementing posting via a custom code(with a proper transaction handling).

In some cases it may be a valid choice, but as we saw file parsing is not a big advantage of DMF and in some cases implementing your own reader can produce better results.

Summary

In this post I provided a sample implementation for a File-based integration for D365FO via DMF. In some ways it is very similar to "Recurring integrations scheduler" but based on X++. It is not complex: the main class that takes a file and passes it to DMF contains about 100 lines of code. Also an advantage of this solution that it can be used as a starting point(for example for prototyping) and then can be easily converted to a custom X++ import when the requirements become more complex

I uploaded files used for this post to the following folder

I hope you find this information useful. As always, if you see any improvements, suggestions or have some questions about this work don't hesitate to contact me.

Monitoring the most problematic performance problem in Dynamics AX - parameters sniffing

Wed, 09 Jun 2021 22:12:03 GMT

One of the complex and probably most common problems in Dynamics AX performance tuning is SQL parameters sniffing. I recently wrote how to fix it, but in this post, I want to focus on a sample approach of how to implement a monitoring solution for such types of problems.

Parameters sniffing example

Let's consider an example from the recent project. One of the critical system processes was a Production journal time posting. After the general optimization, it worked within acceptable time(around 5 seconds) but then on Friday at 3 pm it suddenly slowed down (like 30-50 seconds to post the journal, which affected a lot of users)

TOP SQL for the server looked like this(one TOP1 statement with considerably larger logical reads)

When I checked the plan for this TOP1 statement it was the following

This is a "classic" example of wrong InventSum-InventDim join, where InventDim is used as a leading table.

The solution was quite simple, a custom plan was created with the OPTIMIZE FOR UNKNOWN hint, that ensured that SQL Server would not use first passed location/site values to build a plan

EXEC sp_create_plan_guide @name = N'[AX_InventSumLoc]', @stmt = N'SELECT SUM(T1.POSTEDQTY),SUM(T1.POSTEDVALUE),SUM(T1.PHYSICALVALUE),SUM(T1.DEDUCTED),SUM(T1.RECEIVED),SUM(T1.RESERVPHYSICAL),SUM(T1.RESERVORDERED),SUM(T1.REGISTERED),SUM(T1.PICKED),SUM(T1.ONORDER),SUM(T1.ORDERED),SUM(T1.ARRIVED),SUM(T1.QUOTATIONRECEIPT),SUM(T1.QUOTATIONISSUE),SUM(T1.AVAILPHYSICAL),SUM(T1.AVAILORDERED),SUM(T1.PHYSICALINVENT),SUM(T1.POSTEDVALUESECCUR_RU),SUM(T1.PHYSICALVALUESECCUR_RU) FROM INVENTSUM T1 WHERE (((T1.PARTITION=5637144576) AND (T1.DATAAREAID=N''dsg'')) AND ((T1.ITEMID=@P1) AND (T1.CLOSED=@P2))) AND EXISTS (SELECT ''x'' FROM INVENTDIM T2 WHERE (((T2.PARTITION=5637144576) AND (T2.DATAAREAID=N''dsg'')) AND (((T2.INVENTDIMID=T1.INVENTDIMID) AND (T2.INVENTSITEID=@P3)) AND (T2.INVENTLOCATIONID=@P4))))',
@type = N'SQL',
@module_or_batch = null,
@params = N'@P1 nvarchar(21),@P2 int,@P3 nvarchar(11),@P4 nvarchar(11)',
@hints = N'OPTION (OPTIMIZE FOR UNKNOWN)'

The correct plan after this became the following:

After the hint creation, the posting time returned to normal.

As I wrote in my previous post the insidiousness of such cases is that sometimes people start using reindexing or statistics updates to resolve them, and its slow down the whole system without actually fixing the root of this problem. Such maintenance tasks can lead to Index Maintenance Madness and even Microsoft recently published some notes about this

Let's discuss how we can monitor such issues

Examples from real-life projects

I ran this monitoring on several projects where all Dynamics AX performance tuning was made during the previous audit and got the following results:

Number of queries that need to be analysed was within 100-200 queries. Basically, if you want to support Dynamics AX performance you need to be ready to analyse your system’s top heaviest 200 queries.
Duration of initial analysis - it was about 2-3 calendar weeks, more than I had expected. In this period I was getting several new emails per day
Number of fixed queries - for one project it was 8. 4 queries had an incorrect plan and 4 required new indexes creation. These numbers may not seem high, but keep in mind that previously most performance issues on this project were fixed by the performance audit.

What is really great about this solution is that it gives a certain level of confidence during the Dynamics AX performance optimization project. If a new SQL query appears at the TOP log you will be notified about this.

Some thoughts about D365FO

If you have an on-premise version the monitoring will be the same. For the Cloud version, you don't have a direct access to a Production SQL Server. But the production database has a Query store enabled and information like "TOP queries for the given time" can be obtained using the SAT restore with a read-only flag. The process is described in the following post Using Query Store for Performance Tuning with D365 F&O, but it looks quite complex from a practical point of view(SAT often is used for some other tasks and this restore will take some time)

Probably another option is to implement the same monitoring logic in X++ and run it as a periodic batch job.

Conclusion

Using the described solution you can monitor your Dynamics AX SQL performance and get a notification when some new workload appears in a TOP SQL list. This is extremely useful for the cases when you have random performance problems that can't be replicated on a Test version. You can find and fix exact queries that causing problems instead of wasting time by running reindexing/statistics update.

The code for this can be found in the following folder.

I hope you find this information useful. Don't hesitate to contact me in case of any questions or if you want to share your Dynamics AX/D365FO SQL monitoring approach.

Implementing of Dynamics AX business operations performance monitoring

Thu, 27 May 2021 22:12:03 GMT

Business operations monitoring is used when you have some important processes in your Dynamics AX system and you want to control/measure the performance of them. It is an activity that may be done after the main Dynamics AX performance optimization project and usually it happens when some processes show unstable behaviour. Usually, the reasons for this are the following:

Parameters sniffing
Some external processes that overload hardware

Also, you can get a wrong input from the users, who are complaining that the system is slow, but refering to a wrong process. So having and discussing some numbers instead of general words like slow/fast can greatly improve the communication process.

Performance log implementation detail

I will describe the implementation for Dynamics AX 2012, but probably for all other systems (AX2009 and D365FO) it will be the same.

First you need to create a setup table that defines which Operations to monitor and logging threshold:

Then you create a table that stores performance log data(like Start/End time, Duration, Number of lines, User, etc..)

And you need to write a code that actually creates log data(using a helper class):

    DEV_ProcMetrics     procMetrics = DEV_ProcMetrics::construct();
    ;
    procMetrics.beginTrace();
    // HERE MAIN OPERATION;
    // can use procMetrics.incLineCount();
    procMetrics.endTrace(DEV_BMLogOperation::SalesInvoicePost,
                 int642int(salesParmLineLocal.RecId), // or procMetrics.getLineCounter()
                 "document ref");

Practical examples

Creating the log is quite a simple task, the main difficulty is how to analyse this data to get some valuable insights. And for this purpose Power BI is a great tool, that allows you to view the data from different angles. Let's check some practical examples:

The first one is the case where the users complained about slow production journal posting. To examine this problem we implemented a log and then analysed average posting time. There was some slowness around 1am and it happened almost every day:

Further analysis showed that during this time a complex SQL Agent SSAS cube update job was executed and this slowed down journal posting.

Another interesting case was when the users complained about slow Dynamics AX Sales order posting. This coudn't be replicated on Test system and we implemented a log for 2 main processes that took place when the user pressed a Post button. It was the actual posting and a printing report to a printer. Analysis showed that most of the time system was busy printing documents and this time increased during some hours (like 9am on the graph below)

This allowed us to switch a conversation from "AX performance" to "Network printers performance" and involve proper people to resolve this.

Conclusion

In this blog post I provided some samples on how to implement performance monitoring in Dynamics AX and analyse performance of Dynamics AX operations with Power BI. Sample files(xpo that contains log table, form and a helper class) can be found in the following folder.

I hope you find this information useful. Don't hesitate to contact me in case of any questions or if you want to share your Dynamics AX/D365FO operations monitoring approach.

Improving Dynamics AX2009 / AX2012 logistics performance by denormalizing InventSum table

Thu, 15 Apr 2021 22:12:03 GMT

InventSum table(Oh-hand inventory) is one of the frequently used tables in AX2009, AX2012 and D365FO logistics module. It is also often one of the top reasons for various performance problems due to the structure of the related tables. All quantities and an Item code are stored in InventSum table, but the actual dimension values are stored in InventDim table

A typical system request is to get on-hand data for an item with some dimensions, for example, the quantities of Item AA1 stored on warehouse WW1. And depending on the selectivity of these values, the system may use different execution plans. Very often executing such queries can lead to parameters sniffing and not optimal plans.

A typical resolution is to use literals in the query(the query will be slower and you need to maintain statistics) or to provide a plan hint, but these actions require time and constant monitoring from DBA.

The solution from the cloud

In D365FO Microsoft finally decided to resolve this problem and added all dimension fields into the InventSum table.

So SQL Server doesn't need to decide which table to scan first - InventDim or InventSum, you can apply all filters on one table. You probably need to tune the proper indexes, but it is a less complex task compared to choosing a proper execution plan.

Let's discuss how we can use the same approach to increase AX2009 or AX2012 performance.

Improved version of InventSum table

To add new fields to InventSum table you need to change the following objects:

Object	Change description
InventSum table	Add new fields. Override insert and update methods. Add required indexes
InventUpdateOnhand class	Change updateInventSumSimple and sqlUpdateInventSumStr methods that use direct SQL to update InventSum records
InventSumDelta	Add new fields, Override insert and update methods
InventSumDeltaAggr ..views	These views are used to create InventSumDelta aggregated records
A new job	Create a new job to fill the initial data

(D365FO used SQL Triggers to fill these fields, but probably it is not the best solution)

After that, you need to find places where InventSum is used and change the filter using the following macro.

There is no unified framework for this, so I suggest running a Trace Parser for critical system operations and replacing only them.

Testing the performance change

I tried to test the performance change on the standard AX2012 DEMO VM with the following job that calculates Item quantities per Location:

The VM doesn't have much data and in order to compare the difference I measured the number of logical reads to perform the original(with InventDim exists join) or improved version(just InventSum table) statements

I got the following results:

On the DEMO VM InventSum and InventDim tables have similar size, and even in this case the performance difference is 4x. But if you have a "growing" dimension(like Batch, Pallet, Serial number, LP) and run a periodic InventSum cleanup, InventDim table will have more records and the difference will be bigger. Another important advantage is that filtering by just one table may resolve some parameters sniffing issues caused by joining these 2 tables.

As for drawbacks, you need to create several new indexes on InventSum table. But these indexes will be updated only when a new dimension is created and not during regular quantities updates.

So I don't see any major drawbacks, except the one, that this modification is not a part of the standard AX2012 application(that could have saved thousands of hours spent on various performance tasks by developers, DBA and managers)

Summary

Adding fields to InventSum table may improve AX2009/AX2012 performance and resolve some parameter sniffing issues. It is not a simple, first-priority modification and probably takes a week to implement, but may be considered if you plan to continue to use AX2009/AX2012 application(after the main perfromance audit). Also it is quite important that it uses a solution from D365FO, so it will not cause any problems during the upgrade.

A sample code for AX2012 used in this post can be found here, it may be used as an initial template for a similar modification.

I hope you find this information useful. As always, if you see any improvements, suggestions or have some questions about this work don't hesitate to contact me.

Implementing Dynamics AX 2009/2012 database cleanup

Thu, 11 Mar 2021 22:12:03 GMT

One of the tasks in Dynamics AX 2009/2012 system performance maintenance is performing a periodic database cleanup. In this blog post, I describe a new framework for implementing such tasks and provide some examples.

First, let's discuss why we need a custom code and what are the pitfalls of standard cleanup methods.

A standard way to delete from a large table

There are a lot of standard cleanup procedures already implemented in the system. A good overview for the latest D365FO version can be found here Reviewing Large Tables and Database Growth and Cleanup routines in Dynamics 365 for Finance and Operations

In these procedures you will usually see the following code to perform a "delete from table" operation:

A "while select" loop:

while select MyBigTable
   where MyBigTable.CreatedDateTime < cleanupDate
{
    MyBigTable.delete();
}    
//you need to add ttscontrol here, for example, a commit on every 100th record

This approach can work for small tables, but if we need to delete a lot of records, it takes a lot of time, producing a load to the SQL server.

Another way to implement a delete is by using a delete_from command:

delete_from MyBigTable
    where MyBigTable.CreatedDateTime < cleanupDate;

This statement is fast, but the command delete_from can block the table while running, so you need to run it only during a system downtime.

Both these methods work only per one company, so you need to run a separate cleanup job for each company and that creates a setup overhead.

In a lot of performance audit projects, I asked people why they didn't run standard cleanup procedures and the typical answer was: "We tried to run it a year ago, then it hung, users started complaining about system performance, and we cancelled it".

Improved version of "delete" from a large table operation

So, in order to run cleanup procedures without affecting users, we need a fast and non-blocking way to delete records. Below you can find a sample implementation for this.

IF OBJECT_ID('tempdb..#recordsToDelete') IS NOT NULL DROP TABLE #recordsToDelete
IF OBJECT_ID('tempdb..#temp_hash') IS NOT NULL DROP TABLE #temp_hash
CREATE TABLE #temp_hash (RECID   BIGINT)

declare @step int
declare @isLastStep int = 0;

WHILE (@isLastStep = 0)
BEGIN
    select top 100000 RECID into #recordsToDelete
    FROM [dbo].MyBigTable AS hashtbl            --TABLE HERE
    WHERE hashtbl.CREATEDDATETIME <(GETDATE() -30);

    IF (@@ROWCOUNT < 100000) SET @isLastStep = 1
    CREATE NONCLUSTERED INDEX [##_RECID] ON #recordsToDelete (RECID ASC)

    set @step= 0
    WHILE (@step < 20)
    BEGIN
        SET @step = @step + 1
        TRUNCATE TABLE #temp_hash
        INSERT INTO #temp_hash(RECID) SELECT TOP 5000 RECID FROM  #recordsToDelete;
        IF (@@ROWCOUNT = 0) break;
        --------------------------------------------------------------
        delete FROM [dbo].MyBigTable from [dbo].MyBigTable  AS hs  --TABLE HERE
        INNER JOIN #temp_hash AS JN ON hs.RECID = JN.RECID
        ----------------------------
        delete from #recordsToDelete from #recordsToDelete as dt inner join #temp_hash as dl   on dl.RECID =dt.RECID
    END
    DROP TABLE #recordsToDelete
END
DROP TABLE #temp_hash;

It performs "delete" in 2 steps. First, it selects a large number of table clustered keys(100K) into a temporary table. Often these Keys are a set of RecIds. Then splits them into smaller sets(5K records) and performs the main table "delete" using a clustered key condition.

This approach gives the following advantages:

Delete command is not blocking other processes(as we are using clustered key condition and performing "delete" for a small number of records)
It is much faster than a one by one record delete
It doesn't require additional indexes (by CreatedDateTime) as the number of "large" selects is low
You can stop it in any time without losing the progress
It works for all companies

In my tests the speed of this method was only 40% slower than a standard DELETE FROM command and it can be executed without blocking users even for highly used WMS tables in the middle of the day.

The overall performance depends on many factors(number of indexes/hardware/etc..) but on average it may be 30-40 seconds per 100K records.

User interface to control a cleanup procedure

Let's discuss the typical requirements for a cleanup procedure.

Cleanup consists of several tasks, and we need to specify a cleanup period for each task. Also, it is nice to have some statistics about the last run duration and the number of deleted records.

To control all this I created a "Cleanup settings" form.

Every cleanup task is visible in this form(providing some default values for the settings).

A cleanup task is a class that extends a base class and may implement 3 actions:

Estimate - calculating the number of records to be deleted
Display SQL - instead of executing SQL, displaying it into infolog(used for debugging and validation)
Execute - executing a delete operation

In a class hierarchy it looks like this:

A cleanup class may implement the following logic:

A "delete from a large table" logic described above
Run a standard cleanup class for all companies for complex operations.
Can run X++ code do delete records in order to execute delete() methods and standard delete actions

Example of SQL based cleanup:

Example for X++ cleanup(running a standard On-hand cleanup for multiple companies):

To run these tasks I created a new batch job. You can select one or all tasks using a standard query filter and run them periodically(e.g. weekly). So the whole cleanup process can be implemented with just one batch job, without doing a complex setup of various jobs in different companies.

Progress indicator

Every cleanup task has access to a related record in the Cleanup form and this makes it very simple to implement an execution progress indicator. For SQL based tasks this is done in the base class, for every main loop "Deleted records" field is updated and you can estimate how many records are deleted and how many are left.

Also, and it is very important, you can view last execution statistics for all tasks in one form and this allows you to quickly identify problems with a particular task.

Summary

In this post, I described a custom cleanup framework. If you set up it in a correct way it provides the following advantages:

Fast deletes that don't overload SQL Server and don't block users
Easy to maintain procedure with just one cleanup batch job
Easy to set up and troubleshoot with all settings, statistics and actions in one form

A sample code for AX2009 and AX2012 can be found here, but please note, that it is not a “ready to install and use” solution, all queries should be validated and tested for your environment.

I hope you find this information useful. As always, if you see any improvements, suggestions or have some questions about this work don't hesitate to contact me.

D365FO Performance. Periodic import of one million ledger journal lines

Wed, 20 Jan 2021 22:12:03 GMT

Sometimes you may come accross the following question related to D365FO performance: "Our customer has X transactions per day, can D365FO handle this?". The problem is that currently, Microsoft doesn't publish any D365FO performance tests and the current answer is: "You need to test it with your specific setup". In this post, I try to provide some steps to run a D365FO performance test. It will be for a standard Demo database, so it can be executed without big efforts.

Test description

This test is based on my previous post - How to implement file-based integration in Dynamics 365 FinOps using X++

It loaded 1000 files with 1000 lines each, so 1 million ledger journal lines were created.

To generate such test files I created a new form in the Tutorial integration module:

It allows specifying a link to the Azure file share, a template for a file name, a number of files and a number of lines in each file.

When you press Create ledger journal files button it creates the specified number of files in the Incoming Azure file share folder.

For a test data all possible values combinations for BusinessUnit, Department, CostCenter are used combined with 60* Main accounts for Credit, 50* Main accounts for Debit.

Even in the Demo database, there are more than 1 million different combinations, so all files will have different values for ledger dimensions.

Then I ran a Load messages task as a batch job and measured its execution time. It created and posted 1000 ledger journals(with 1000 lines each).

Test setup and preparation

To run this test I used a standard cloud DEV VM with 16HDD and the Demo database.

I did the following preparation:

Installed DEVCommon and DEVTutorial models from https://github.com/TrudAX/XppTools repository.
Created a new Azure file share and created a ledger message type linked to it

Switched off DEV VM monitoring. It was creating a lot of load to a file system for my already not quite fast VM.

K:\Monitoring\MonitoringInstall-Uninstall.ps1

Disabled Budget control. You probably will not use it for such large data loading.

Budgeting - Setup - Budget control configuration

Adjusted a Ledger account structure setup to allow any Business Unit, Department, Cost Center values for used Main accounts.

Set Maximum batch threads to 60. This allowed me to fully utilize my VM.

After that I generated 1000 files and ran the test.

Test results

I used a standard cloud DEV VM with 16HDD and the Demo database. I ran the test on 2 VM sizes:

Standard D8 v3 (8 vcpus, 32 GiB memory)
Standard D16s v3 (16 vcpus, 64 GiB memory)

For each VM size I decided to perform 2 tests:

I started with a fresh Demo database. Loaded and posted 1 million records, then generated the same set of files and loaded them again for a different date. So totally 2 million journal lines were created. The first test generated a new ledger dimension combination for every line, so it was like an initial data load. The second test used existing dimensions combinations, so it was like a periodic load.

The load operation consists of 2 batch jobs. The first reads files from Azure and puts them into D365FO tables. It is a single-threaded job and for both tests, the time was around 1.5 minute for 1000 files. Then the processing job(which is multithreaded) reads the file content from D365FO tables, generates and posts ledger journals.

I got the following times for loading and posting 1M ledger journal lines:

VM type	First load time	Second load time
Standard D8 v3	4h 40 min	3h 40 min
Standard D16s v3	3h 6 min	2h 29 min

During both tests, CPU was used for 100%(about 25% for SQL Server and 75% for Batch service). Ledger journal classes contain a lot of caching and probably they are not optimized for different values for every single line as in this test.

Of course, this test is quite simple, you should not use results from Tier-1 machines for Production planning, as Tier-2 VMs have a different architecture(SQL Azure instead of SQL Server), but I think the behaviour for Tier-2 will be the same: the more expensive service you use, the faster system will work.

I don't have access to Tier-2 VM now, also it is not clear what level of resources Microsoft gives you for Tier-5(Performance) and Production and how these resources operate comparing to local DEV boxes. If you have any information regarding this, please share it with us.

Summary

The cloud has some amazing capabilities. Even in this test by just one click(changing a VM size) I increased the load performance by almost 50%. But everything has its price. The current Dynamics 365 Licensing Guide has more than 50 pages, so the pricing structure is quite complex. And probably in the cloud world instead of asking "Can the system load 1M journal lines?" you should ask "What will be the cost of loading 1M lines in 2 hours?".

You can use the provided example for performance testing of File-based integration for D365FO. I uploaded files used for this post to the following folder.

I hope you find this information useful. As always, if you see any improvements, suggestions or have some questions about this work don't hesitate to contact me.

How to implement file-based integration in Dynamics 365 FinOps using X++

Thu, 17 Dec 2020 22:12:03 GMT

Update 2025-02-04: The integration was updated to a new Azure Storage DLL A new way of working with Azure File Share from X++

There are many ways to implement integration with Dynamics 365 for Finance and Operations. One of the comprehensive descriptions of possible options can be found in this great post by Tayfun Sertan Yaman: How to integrate with Dynamics 365 for Finance and Operations.

The file-based approach is probably the oldest and commonly used integration type. Using this approach, an external system and D365FO send messages by reading/writing files into some shared network folder. It has the following advantages:

Very clear responsibility (you either have a file in the correct folder or not);
It is easy to troubleshoot (you can view the file, can modify it, etc..);

A good overview of some options how to work with files in D365FO can be found in the following video by Ludwig Reinhard Mass importing data in D365FO. He describes the pros and cons of two solutions:

Recurring integrations scheduler(RIS)
LogicApp package

Both these solutions are valid for a lot of cases, but they have the following issues:

You need to install/support additional software(Cloud-based LogicApp or On-premise RIS)
They use the Data Management Framework to load files, that doesn't support transactions. This can be critical for errors processing.

In this blog post, I try to describe another possible solution(with "Consuming external web services" type) for the File integration using X++ that is resolving these issues.

Solution description

Let's consider that we need to implement a periodic import of files and create ledger journals in D365FO based on these files. They have the following structure:

A company where ledger journals should be created is defined in a file name(text before "_") and one file represents one journal.

As the result we should get a posted journal in that company:

The integration will be based on two of my previous posts: How to read CSV/Excel files and How to create ledger journals in X++.

In order to copy files to Azure storage from the local PC the following command may be used

azcopy copy "C:\AAA\CCC\Files/*" "https://f365vmstorage.file.core.windows.net/ledgerinterface/TestUpload/SASToken"
(if used in bat file need to replace % to %% in SASToken value)

Proposed solution

In the following section, I provide some code samples that can be used as a starting point to implement a periodic file import and processing.

As D365FO is a cloud-based system we need to put our files to some network location that is accessible from the cloud. Probably the simplest way to achieve this is to use Azure file share. It can be created with several clicks from Azure portal and you can even map it as a network drive in Windows(or use Azure Storage Explorer).

For this post I created a new account with the following folders:

To store connection details I created a simple form in D365FO

Notes:

Another alternative to Azure file storage can be an SFTP site. Check this article that describes this in details: Read File’s from SFTP Server and Write data in AX365. Update: (thanks to Tommy Skaue and Volker Deuss): SFTP connection doesn't recommended for new deployments: https://docs.microsoft.com/en-us/dynamics365/fin-ops-core/dev-itpro/deployment/known-issues-new-deployment-experience#ftp.
Instead of storing Connection details in a D365FO table,a more secure solution is to create an Azure Key Vault and put all secrets into it(for example like this) and store a reference to this Key Vault in our new Connection table.

Inbound message types form

Next form to describe our integration will be Inbound message types form

This form contains 3 main sections:

1 - Details tab

Defines Incoming and Archive folders in our File share. There will be no Error folder: if an inbound file fails validation then the error details will be found in the message table.
Contains link to the Class that will do processing from this folder. The class should extend a base class DEVIntegProcessMessageBase and implement the following method:

abstract void  processMessage(DEVIntegMessageTable  _messageTable, DEVIntegMessageProcessResult _messageProcessResult)
{
...
}

This method will be called by the integration engine outside of a transaction, so all transaction control can be implemented per Message type. There can be different options here: one transaction per message, multiple transactions(for example if a file contains several independent journals) or a single transaction per line. The result of processing and the current stage should be written to _messageProcessResult variable, so in case of an unhandled exception, this information can be saved for review. Also, this class will be created one time per full import session, so it can implement different caching options.

2 - Operation parameters tab

Contains parameters that are individual for the current operation. In our case it will be:

Ledger journal name reference for journal creation
Post the journal(No/Yes) and
A file type(Excel or CSV)

3 - Advanced settings tab

Contains some common parameters: If we should use Parallel processing for our incoming files and how to move files to an Archive folder(with the same name or append DateTime to the file name). Parallel processing is based on this post: A simple way to implement a parallel batch processing in X++, so for example if we set it to 10 and have 1000 incoming messages, 10 batch threads with 100 messages each will be created.

Also, this form contains two servicing operations:

Check connection button that tries to connect to the specified directory
Import file button that can be used in testing scenarios to manually import a file from a user computer without connecting to network file share

Incoming messages form

This table will store the details on each inbound file.

Every message has a status field that can contain the following values:

Ready – a file was read to D365FO and successfully moved to the Archive folder.
Hold – The user has decided not to process the file. This is an alternative to a delete option
In process – system-generated status, a message is processing now
Error – failed validation
Processed – completed successfully

In this form it is also possible to do the following operations:

View incoming file context
Filter by different statuses
View a detailed error message
Change the status to process the message again
View file processing statistics (processing duration, time, number of lines)

Load incoming file operation

It is a periodic batch job that we can run for one or multiple message types.

It connects to the shared folder, reads files, creates a record in Incoming messages table with Ready status, attaches a file content to this message and moves the file to an Archive directory. If Run processing is selected, after the load system will execute processing of the loaded messages.

Process incoming messages

Message processing may be executed as a separate operation - Process incoming messages that selects all not processed messages and calls the processing class for them.

The logic of how to process the file is different per message type/class. For the simple scenario, the class can just read the file content and create some data in one transaction. For this blog post, I implemented two step processing. See the sample diagram below:

During the first step, the class reads the file and writes data into a staging table. A sample code for this:

    while (fileReader.readNextRow())
    {
        linesStaging.clear();
        lineNum++;
        linesStaging.LineNumber     = lineNum;
        linesStaging.HeaderRefRecId = tutorialLedgerJourHeaderStaging.RecId;
        linesStaging.MainAccount  = fileReader.getStringByName('MainAccount');
        linesStaging.BusinessUnit = fileReader.getStringByName('BusinessUnit');
        linesStaging.Department   = fileReader.getStringByName('Department');
        linesStaging.CostCenter   = fileReader.getStringByName('CostCenter');
        linesStaging.Amount       = fileReader.getRealByName('Amount');
        DEV::validateWriteRecordCheck(tutorialLedgerJourLinesStaging);
        tutorialLedgerJourLinesStaging.insert();
    }

Then based on this staging data values, a new journal is created. As I wrote in this post there are two options to create a ledger journal: either using LedgerJournalEngine class or using a data entity. The choice between these two should be made by answering the question: if the user wants to create the same journal manually, does he use manual entry or data import?. In this case, I want the result to be similar to manual entry, so LedgerJournalEngine class is used.

ledgerJournalTrans.AccountType          =   LedgerJournalACType::Ledger;
ledgerJournalTrans.modifiedField(fieldNum(LedgerJournalTrans, AccountType));

DimensionDefault  dim;
dim = DEVDimensionHelper::setValueToDefaultDimensionCon(dim,
  [DEVDimensionHelper::BusinessUnit(), tutorialLedgerJourLinesStaging.BusinessUnit,
   DEVDimensionHelper::Department(),   tutorialLedgerJourLinesStaging.Department,
   DEVDimensionHelper::CostCenter(),   tutorialLedgerJourLinesStaging.CostCenter ] );

ledgerJournalTrans.LedgerDimension = LedgerDimensionFacade::serviceCreateLedgerDimension(
  LedgerDefaultAccountHelper::getDefaultAccountFromMainAccountId(tutorialLedgerJourLinesStaging.MainAccount), dim);

ledgerJournalTrans.modifiedField(fieldNum(LedgerJournalTrans, LedgerDimension));
ledgerJournalEngine.accountModified(LedgerJournalTrans);
....
ledgerJournalTrans.insert();

After the journal creation, this class runs journal posting.

Error types and how to handle them

It is not a big task to create a journal based on a file. The complexity of integration is often related to exception processing and error monitoring. Let's discuss typical errors and how users can deal with them.

File format errors

The next error type is a wrong file format, so we can't even read the file content.

To test this case I renamed one of the columns

After the import users will see this file with the Error status. Notification can be done using standard filtering by the Status column.

Users can view the error log, then download the file and check the reason for this error. There may be two reasons:

Our code that reads the file is wrong. In this case, we can send this file example to a developer to fix the logic. After fixing the problem we can run the Processing again.
External system sent a file in the wrong format. In this case, the user can send this file back to the external party, then change the message status to Hold.

Data errors

The file has a correct structure but contains a wrong data(e.g.. values that don't exist)

In this case, a Status of our Message will be Error and an Error log will be generated.

Users can view this error, display a Staging data to check the values from the File and take some actions(e.g. create missing values in the related tables if they are valid). After that, they can Process this message again.

In some implementations(EDI), we can even allow staging data editing.

Posting errors

A similar type of error is a posting error. For example, in a current implementation if the journal is not balanced the error will be generated and the message gets the Error status:

A possible variation to this approach is to create a document(journal in our case), try to post it, and even if posting fails, still set the message Status to Processed and leave the journal unposted, allowing accountants to decide what to do with it. As we don't process in transaction this will be a simple modification for our process class.

Wrong result errors

That is probably the worst scenario. The file was processed successfully, but the resulting journal contains some wrong transactions.

To analyse the result, users can view the staging data and check that they are correct

Another useful option to troubleshoot this scenario is a parameter in the Message types table for our operation: Post journal(No/Yes). We can switch it off, manually load a test file and check the created journal without posting it. And that may give an idea of what is wrong.

Summary

I provided a sample implementation for a File-based integration for D365FO. The main concept of it is to create a basic framework to simplify troubleshooting(most typical errors and all related data can be viewed in one form - Incoming messages) and provide some additional logging.

This may or may not be appropriate in your case(there are different options how to implement this). Anyway I recommend to use the following checklist while designing the integration: Integration solution specification

I uploaded files used for this post to the following folder

Another important question when you implement a solution like this: is how fast will be your integration. I wrote about sample steps for performance testing in the following post: D365FO Performance. Periodic import of one million ledger journal lines

I hope you find this information useful. As always, if you see any improvements, suggestions or have some questions about this work don't hesitate to contact me.

Analysing Dynamics AX / Dynamics 365FO batch performance using Power BI

Wed, 25 Nov 2020 22:12:03 GMT

Batch task processing plays an important role during Dynamics AX / Dynamics 365FO performance analysis. Batch tasks are executed on a separate server but may affect system performance producing a load to SQL Server database. In this post, I show how to use Power BI to analyse batch performance.

Getting the data

The source data for our report is a Batch execution history table(BATCHHISTORY) where the system logs all batch runs(you need to check that "Save job to history" flag is enabled for a batch job).

In a lot of cases, you can't install an additional software on a SQL Server to query this data directly. So the easiest way to get the required data is to execute a query in SQL Management Studio and copy/paste its result into the local Excel file. Then this file can be used as a source data for our Power BI report.

See below the sample query that I use for AX2012

DECLARE  @m int
set @m = DATEDIFF(mi,SYSUTCDATETIME(), SYSDATETIME())
SELECT
CASE
  WHEN [BATCHHISTORY].[STATUS] = 3 THEN N'Error'
  WHEN [BATCHHISTORY].[STATUS] = 4 THEN N'Finished'
  WHEN [BATCHHISTORY].[STATUS] = 8 THEN N'Canceled'
  ELSE CONVERT(nvarchar, [BATCHHISTORY].[STATUS])
END as [STATUS]
      ,[BATCHJOBHISTORY].[CAPTION] as [BatchCaption]
      ,[BATCHHISTORY].[CAPTION]    as [TaskCaption]
      ,[BATCHHISTORY].[BATCHJOBID]
      ,[BATCHHISTORY].SERVERID
      ,(SELECT TOP 1 [Name] FROM [DynamicsAXProd_model].[dbo].[ModelElement] where axid = [BATCHHISTORY].CLASSNUMBER and ElementType = 45) as ClassName
      ,[BATCHHISTORY].CLASSNUMBER
      ,DATEADD(minute, @m, ORIGSTARTDATETIME) as [ORIGSTARTDATETIME]
      ,DATEADD(minute, @m, [BATCHHISTORY].[STARTDATETIME]) as [STARTDATETIME]
      ,DATEADD(minute, @m, [BATCHHISTORY].[ENDDATETIME]) as [ENDDATETIME]
      ,DATEDIFF(ss, [BATCHHISTORY].STARTDATETIME, [BATCHHISTORY].enddatetime) as [DurationSec]
      ,[BATCHHISTORY].[COMPANY]
      ,[ALERTSPROCESSED]
      ,[BATCHCREATEDBY]
      ,[CANCELEDBY]
      ,DATEDIFF(mi, [BATCHJOBHISTORY].ORIGSTARTDATETIME, [BATCHJOBHISTORY].STARTDATETIME) as [StartDelayMin]
  FROM [dbo].[BATCHJOBHISTORY], [dbo].[BATCHHISTORY]
where
[BATCHHISTORY].BATCHJOBHISTORYID = [BATCHJOBHISTORY].RECID and
DATEADD(minute, @m, [BATCHJOBHISTORY].[STARTDATETIME]) > CONVERT(datetime, '2019-08-12', 120)

You need to adjust STARTDATETIME filtering, limiting the number of rows to 100-200k(start with one previous week for example). For D365FO you can also create a data entity for it or just run this query for a PROD database copy.

Batch analysis reports

Let's see what kind of reports we can build from this dataset:

- Total duration

The first report we can build analysing Total duration of batch tasks from different measures

The first visual here displays Total duration by task name, where we can find what are our longest tasks and then invest our time into optimizing top tasks from this list.

The second visual displays how busy our batch servers were(Duration by ServerID). In the example above we can see that the amount of work executed on AOS01 is way less than on AOS02, so maybe it is a good idea to relocate some tasks to it.

And the third visual displays the number of batch tasks and their duration by company, so we can define our top companies.

- Delayed batch tasks

There can be a situation where a batch task is scheduled for a particular time but was not executed on this time because there were not free threads available. We can calculate the delay as the differences between StartDateTime and OrigStartDateTime columns.

One of the main parts of configuring a batch server is to specify a "Maximum batch threads" that it can handle. The situation where you have delayed tasks is not normal, on a properly configured system this should never happen.

From practical experience: don't set this value too low. A modern 8-core server can handle 50-200 batch threads, so start with this interval. There can be some specific batch jobs that mostly use AOS resources and no SQL work, only in this case it may be set lower. Check for % of AOS CPU usage.

The second report displays such delayed tasks.

It allows identifying which tasks were delayed, on what server and at what time. In the example above we see that there were a lot of delayed tasks around 9 pm. It was a project case where we have found that "Customer ageing report" was incorrectly configured and that overloaded batch server with multiple tasks. If you see some custom jobs here maybe you need to review your batch strategy

Batch tasks schedule

Another important part of AX performance analysis is analysing how batch tasks are scheduled during the day. Usually, it is not a problem if a large task is executed at night, but if it runs during the day it may affect users' work.

The third report displays how the top batch jobs were executed during the day.

We see that there are 2 big tasks executed after 7 pm and one big task executed around 1 pm. Then it is worth discussing whether we need it in the middle of the day and how we can optimize it.

Conclusion

With the help of Power BI you can analyse 3 main points of Dynamics AX/ D365FO batch tasks performance: Longest tasks, Delayed tasks and Schedule during the day. I upload files used for this post to the following folder

I hope you find this information useful. As always, if you see any improvements or suggestions, don't hesitate to contact me.

Comparing D365FO development VMs performance(VS2017)

Wed, 21 Oct 2020 20:12:03 GMT

Starting from PU37 Microsoft has changed Dynamics 365 Finance and Operation development environment from Visual Studio 2015 to VS2017. This post is an updated version of my previous one for VS2015, where I try to compare the performance of 4 development configurations with the new VS2017 version.

I have chosen the following VMs:

Local Hyper-V Image
Standard VM based on 3 HDD disks - this is a default Development setting in LCS
Azure VM based on Premium SSD disks
Azure VM based on 15 HDD disks - changed via LCS Advanced settings

Detail specifications of these VMs:

Name	Local	SSD	HDD15	HDD3
VM Type	Local Hyper-V Image	Standard D8s v3	Standard D12 v2	Standard D12 v2
CPU	Core i7-8700 3.2GHz, 6 cores, 24 GiB memory	Xeon Platinum 8171M 2.10GHz, 8 vcpus, 32 GiB memory	Xeon Platinum 8272CL, 2.60GHz, 4 vcpus, 28 GiB memory	Xeon Platinum 8171M 2.10GHz, 4 vcpus, 28 GiB memory
Storage	Samsung 970(more than 100k IOPS)	3 premium disks 512GB (2300 IOPS) + build	15 HDD disks 32GB (500 IOPS)+ build	3 HDD disks 512GB (500 IOPS)+ build
Run cost	Box for run 3 VMs - around 1.5k$	0.75$ per hour	0.52$ per hour	0.52$ per hour
Storage cost (monthly)	0	219 USD	24 USD	65 USD

One thing to note: in Azure the performance of SSD disks depends on the disk size and it is different from HDD disk performance that is always 500 IOPS.

That means that in Azure world SSD disk is not always better than HDD. It is also tricky to do any tests with an SSD because they have disk bursting that can temporaly increase disk IOPS to 3500 up to 30 minutes. I did some testing with 16HDDs(500 IOPS) and 3 SSDs 512GB, 2300IOPS max( + Build 26GB ) and got the following results: for 16HDD you will get 8000 stable IOPS, for 3 SSD you get 7100 stable IOPS and Bursting can temporarily add 3500 IOPS.

System version and initial setup

Both 4 VMs have used the same D365FO version: "Finance and Operations - Develop (10.0.13 with Platform update 37)". After installation I disabled the following services:

Microsoft Dynamics 365 Unified Operations: Batch Management Service
Microsoft Dynamics 365 Unified Operations: Data Import Export Framework Service
Management Reporter 2012 Process Service

and added D365 Defender Rules:

Install-Module -Name d365fo.tools
Add-D365WindowsDefenderRules

Overall system performance tests

First, let's compare overall system performance by running a full compile and full DB sync. They are not frequent operations for Dev machines, but they are very resource-intensive and it allows us to find out VMs overall capacity.

For the compile task I got the following results:

The compile times are longer than in previous test with VS2015 and 10.0 version.

Next test is a database synchronize:

Synchronize operation is an IO and CPU-intensive task. Local VM has a more powerful IO system, and a more powerful CPU, that explains a better result. SSD VM performed better compared to HDD in this test due to faster disks. And again for all VMs there is ~30% performance decrease compared to VS2015.

Daily task tests

To test performance for more frequent developer tasks I chose 2 tasks: time to hit breakpoint and time to display 'Hello world' message from the job. In AX2012 both these tasks have a near-zero execution time, so you don't need to wait.

Time to hit breakpoint test

Before the test I opened the D365FO main screen, to warm up the system cache.

To prepare for this test I switched off 'Load symbols for items in the solution'. Then I opened AOT, searched for a SalesTable form and added it into the new project and marked the form as a startup object. After that I opened the code and added a new breakpoint to the init() method.

Time in this test is the time between I pressed Start and the time when the breakpoint was hit.

For HDD03 VM this test initially failed(I got a Timeout message).

Two tests were performed to see how the cache would change this time.

The result probably depends on IO speed.

Hello world test

This test was performed straight after the breakpoint test, but I restarted Visual Studio. In this test, I created a new project and added a Runnable class with the following code

info("Hello world");

On the second run, I changed this text to "Hello world2". On the Third run I didn't change the text, just ran the same job.

I measured the time between pressing Start and the time when the message displayed in the browser.

Here are the results:

The problem is that for HDD based machines the test failed in the first attempt, Visual studio just hung. The solution was to kill the process and restart it again(that explained that HDD "First times" are better)

Update 10.0.37 local

Recently I made an upgrade for my local PC and tested the performance of the local 10.0.37 VM with VS2022. I upgraded to a Ryzen 7 7700X CPU(not the fastest, but close to the top CPU https://www.cpubenchmark.net/singleThread.html), and a Samsung 990PRO SSD(more than 1 million IOPS). The results are the following:

Full sync for the database: around 3 minutes
Recompile the project: almost instant (1.5 seconds)
Refresh the simple form (like CustGroup) after recompile: 25 seconds
Refresh the complex form (SalesTable) after recompile: 35 seconds
Put a breakpoint to SalesTable.init and attach to process: almost instant (2-3 seconds)

Local VMs may not be the most convenient options, but if you write a lot of X++ code(with change/test/fix cycles), they are worth considering.

To increase response time even more, enable Preload feature https://abhimantiwari.github.io/blog/Application-Initialization/ (thanks Huber Gomez Hernandez for this advice)

Update 2024-10-17

Make sure you disable the "Build metadata cache when AOS starts" parameter in the System parameters form(System administrator → Setup → System parameters)

https://www.linkedin.com/pulse/boost-your-x-development-vm-performance-2x-denis-trunin-krowc/

Enabling disk cache also adds some improvements. With Read it takes about 60 seconds to refresh the browser after rebuild

Conclusion

Let's summarize the current recommendations based on these tests for Visual Studio 2017 D365FO development environment and what to do if it is slow:

If you want maximum performance and can buy a new hardware(and now you can buy a PC that has 10-20% more speed than mine), Local VM is the best choice, but this VM is hard to manage.
Standard Development Environment from LCS(with the default of 3 HHD disks) is the slowest and constantly hangs. If you can control the Environment creation, never use it and always change the number of disks(15 32GB HDD are cheaper and faster).
HDD15 VM with ..V5 CPUs is the best choice in terms of price/performance. It is a little bit slower than SSD based VM, but the storage price(that should be paid even if VM is deallocated) is almost 10 times cheaper(24 vs 220 USD)

I will also try to repeat the same tests after the next D365FO release, it will be interesting to compare the progress in this area. Any comments are welcome.

How to manage a Dynamics AX performance optimization project

Wed, 07 Oct 2020 22:12:03 GMT

Your Dynamics AX system(AX2009/AX2012 or Dynamics 365 for Finance and Operations) is slow, the users are complaining about performance and it is not clear what to do to fix that. The solution is to start a performance optimization project. How to organize this project, who should be involved, and how the process should be managed is explained in this blog post.

We recently finished quite a successful performance optimization project and I will use it as an example for this post. The similar approach described here was used on dozens of different AX2009/AX2012 projects. I also describe common mistakes to avoid.

Project overview

The client was a big retail company running AX2009 for about 10 years with 80 users and 1TB database size. With the COVID-19 situation, they saw an increase in their sales and the performance of Dynamics AX, that had been average before this, became critical.

As a final result, they saw a great improvement in AX performance.

System stability was greatly improved, here are some measurable results:

In total, we did 43 different performance tasks and the project length was 3 months. So these results were quite amazing.

As an AX technical consultant, I can say that the project flow and organization were great and contained all typical interaction types that exist in similar projects. The project size due to system complexity was above average(for example Technical consultant time was 45 days, that is more than usual).

I want to describe the pieces of success and common mistakes that I saw on other projects like this.

Performance project team

The main factor for success is the proper project team. A typical project team should have the following people:

IT manager
Functional consultant
Infrastructure manager
SQL administrator
Project manager
AX Technical consultant

One person can combine several roles but all these roles should be a part of the project.

Typical mistakes of organizing a team

Let's discuss what can happen if we don't configure a team as recommended:

- No project manager allocated

Initially that seems reasonable: the system is critically slow, you need just technical people who tell you what to do and not managers. But this became an important factor later.

A lot of performance tasks are complex, have no direct impact on users and require efforts from different people to be solved.

Probably the most typical example here is a recurring batch job that tries to reserve quantity for open orders, but some orders are old, will never be reserved and the process happens again and again. Or an integration task that tries to process wrong messages every time it runs and does not mark them with an error flag.

Such jobs can produce a huge constant load on AOSes and SQL Server and are quite complex to resolve. You need to contact business users, understand the reason for such multiple processing, find a solution, develop and test a fix(this can be complex for a procedure that was developed 5 years ago and nobody wants to touch it). So a good project manager should allocate resources for tasks like this and control the execution.

- Only SQL DBA/Infrastructure team is involved

This is a typical situation in the beginning. Users complaining about performance, you ask you DBA to check servers and find the reason. Often this doesn't work and ends up with more statistics or index updates. A lot of performance problems are related to parameter sniffing issues or Dynamics AX settings/code, and it is hard for SQL DBA to fix that, without knowing what is the logical purpose of these tables and what should be the correct SQL plans for long queries.

- SQL DBA/Infrastructure team is not involved

This is the opposite situation to the previous case. Sometimes performance problems are related to SQL Server or hardware configuration.

This tweet probably represents a common problem. Hardware used for multimillion ERP implementations is less powerful than a typical gaming laptop.

If you don't include SQL/Infrastructure team into the project team, there is a high chance they just push back any recommendations.

Even for the most obvious performance counters, it is quite easy to reject recommendations, let's take a CPU load as an example. If the load on AOS is 80%(that is really big) and the recommendation is to add more CPU cores, it can be argued that almost 20% are not used and everything is OK. Or if the CPU load is 100% for 10 minutes and then 20% for 10minutes - the average will be 60%. Such discussions are very time consuming and not productive, and may be simplified if all participants are within one team.

- Only AX technical consultant involved

Your Dynamics AX is slow and you ask AX technical consultant to check why and give their recommendations. A typical example of this: a company hires a Microsoft Services consultant, they do a 3 days analysis with DynamicsPerf tool and at the end give you a very nice looking report.

This may solve simple issues, but may not work in some difficult cases.

For example, for this project one of the problems was related to slow shipment processing. When we started analysis we have found that a shipment module was using a lot of cross-companies queries, but the actual company was always defined in the business process, so these cross-companies queries were removed and it gave quite a considerable performance boost.

Finding and fixing issues like this involves a lot of communications and may not be resolved just by a single person sitting and running some queries.

The Project flow and tasks

The team should meet on a very regular basis(like once a week) to discuss what changes and tasks should be done in the system to make it faster.

Quite an important point here is that each task should be testable, tracked and have a status. Don't allow email chains like in this screenshot, where one problem was being discussed for 2 months with more and more people getting involved.

The initial set of tasks usually comes from a Performance audit, here is a "typical" example:

Other tasks come from 2 sources:

User complaints (e.g. an operation works slowly)
Periodic servers monitoring(Top SQL queries, Blocking, missing indexes, etc..)

For user complaints, the first question that should be asked is "Can we replicate this issue in a Test environment?". If we can do this it means that the task is quite simple, we allocate it to a developer for the tracing and investigating. If the answer is "No" then the team should discuss the plan to proceed. Probably the first thing to do in this case is to implement a tracing solution that will allow us to operate some numbers(e.g. how often the issue happens, at what time, what are the delays and so on) rather than just general "the system is slow".

Also, before discussing any optimizations it is worth finding what business task users are trying to resolve, maybe there is another more efficient way to do it.

The classic example here is a case where the whole accounting department complained about slow Dynamics AX2012 performance. The Client had already upgraded SQL hardware, installed a new AOS, but nothing helped. When we discussed the issue we found that a part of users daily work was to update clients information. In order to do this, they opened "All customers" list page, filtered it by the required customer, double clicked on it to open "CustTable" form, edited the information, then closed CustTable form, filtered "All customers" list page again by the next customer and so on... "CustTable" is a very heavy form that takes about 5 seconds to load and such delay was very annoying if you need to update 20-50 customers. The solution was quite simple: we just showed them that they don't need to close "CustTable" form to switch the active customer, there is a Grid view in this form where you can almost instantly search for it. The users were very happy.

Ready to change and forward only approach

That is what I suggest to discuss with a customer at the beginning of the project. The only way to fix performance problems is to change something. There is no magic flag "Run faster" in Dynamics AX or SQL Server.

These changes can be done in multiple areas, but as with any change, there is always a chance that it can affect the system in a negative way. We try our best to avoid this, but such issues are happening on almost every project. Even a positive change(you created a new index or cleaned up some tables) can produce a negative effect due to parameters sniffing.

The only correct way to deal with such issues is to continue the system monitoring, identify and fix them quickly. Not a correct way is to perform a rollback.

Some practical recommendations:

Have someone from the customer team to apply any changes to Production. There were a lot of situations like this: you created an index and the next day some users can't print documents(totally unlinked events). Then there were usually a lot of emails trying to find the reason and blame someone and the person who did the last change is a perfect candidate for this.
Allocate a day to monitor the system after the changes were applied. This is needed to catch and fix parameters sniffing issues.
For the first round of changes include as many as possible. The strategy "one change in a time" doesn't work well in the beginning. Often the first positive feedback from users creates a lot of trust, that makes the project flow simpler later.

Prepare a separate LAB version for the project

Often the most complex problem in performance optimization is to replicate an issue. There are a lot of cases where the operation(for example Sales orders posting or some batch job) is working without any issues during the day, but becomes slow during a certain hour. The reason may be in other parallel processes that caused blocking or the high system load, but very often this depends on the particular data used in this process.

Typical example: warehouse operations where for some period you can have zero lines ready for shipment, but an hour later there are 10k lines.

To trace such issues quickly, a database point-in-time restore(like restoring the Database at 11.32 am) can provide valuable information. So organizing a separate environment where such backups can be restored and traced can save a lot of time allowing to quickly replicate an issue and test the fix later.

You don't need to replicate full Production hardware for this. In this project, we used a separate one-box environment(that included AOS, SQL and all other components) with 8 CPUs cores, 48GB of memory and 3TB drive(to keep 2 copies of database backup). Also, an important tip is to run this environment under a user that doesn't have any Production access to avoid situations of sending e-mails to the real customers or to the production integration folders.

Deployment and communication channel

This process of Dynamics AX performance optimization is iterative: we fix first top current issues, then continue monitoring, provide a new set of actions and so on... It is quite important to minimize the time that is needed to deploy the changes to Production. For some customers, it is not a problem, but may be challenging for others.

I suggest reviewing the deployment process and try to simplify it if possible. Ideally, we should be able to do deployments every day if needed.

Also, a great role in project success plays a good communication channel(other than e-mail). A group in Microsoft Teams is probably the best solution for this, however, sometimes it is tricky to create it due to different users' domains(check this trick if you want to run multiple Teams on one computer).

Deal with external integrations

Some integration types can often cause performance problems. In AX2009/AX2012 such "dangerous" types are:

SQL access to the Dynamics AX database for the 3-party application.
AIF or Service endpoint to query some information

The common problem is that these 3-party applications can easily stop the whole system by calling these services or running SQL in an uncontrolled manner. The main problem here is that you can’t control or change these 3-party applications, they often belong to a different team that doesn’t care about ERP problems.

Examples that we had: SQL query to get customer information(Address, Contact info) with some missing DataAreaId field in joins, that leads to full table scan for each call, or frequently calling AX service that returns On-hand data for the whole store.

In cloud D365FO version, Microsoft faced exactly the same problems(OData replaces SQL access for the cloud) and the solution they proposed is a Priority based throttling. That means when the system is highly loaded such queries will get an exception. There were some negative comments about this feature on Yammer, but when you see how the whole Dynamics AX performance is affected just by one incorrectly written query, your opinion may be changed. So throttling can help in this case(at least start the dialog about optimization).

The best solution is probably not using such integration types if you can't control the external system. If you need to provide some information to the 3-party system, just unload it to the separate database(like BYOD in Dynamics 365FO).

But it is very hard to change the integration process that is already implemented, so a typical approach is optimizing X++ code related to the service, and discussing call frequency with the external team(that is where you need an "IT manager" person).

Conclusion

So to deal with Dynamics AX performance problems you need the following:

Organize a team with the right people
Properly manage performance issues
Prepare a test version
Do a good integration design

This allows you to quickly resolve and fix any problem.

I hope you find this information useful and will use it in case of any AX2009, AX2012, Dynamics 365FO performance troubleshooting. As always, in case of any problem, suggestion or improvement, do not hesitate to contact me, I will be happy to discuss it.

Update 08/11/2020 - check this article from Glen Turnbull about D365FO specific tasks Managing General Performance Issues in Microsoft Dynamics 365 Finance and SCM

Execute direct SQL in D365FO database

Sun, 02 Aug 2020 22:12:03 GMT

'Execute direct SQL' is a simple form that allows to write and execute direct SQL from the browser on D365FO database. It can save time when you are debugging or troubleshooting.

The original idea was introduced in AX Paradise blog post, I added some improvements and included it to my list of tools.

How to work with this form

To use this form, you need to go to Administration-Inquiries-Database, and run SQL Execute form.

It allows you to enter SQL text to execute and outputs the execution results as HTML or as a File when you press the Run button.

You can limit the number of returned rows. Due to AX string manipulation and text formatting current output is quite slow, it can handle only about 100-1000 rows.

As direct SQL execution is quite a dangerous operation, I also added a second tab to this form that logs all executed commands, so you can always check who used it.

To limit the number of users who can use this tool I added a new role - DEVSQLQueryExecute(it is required even for SysAdmin)

Performance inquiry

One of possible use cases of this tool can be a performance troubleshooting for D365FO. Current tools for this in LCS are quite slow for a cloud version.

You can execute commands like getting TOP SQL without a direct connection to SQL Server.

To return large text or the result of UPDATE statement you may update a SQLResults field(as standard ResultSet has some limits, so large text can't be returned directly).

For example, you can also download and analyse SQL plan using the following command:

However, I don't know whether this will work for production instances(due to security rights). If you can test it, ping me with the results. If it doesn't work, probably we should create an Idea to allow this(at least until LCS views will be fixed). For a list of sample queries check Glenn Berry's monthly update of Azure SQL Database Diagnostic Information Queries

Summary

Execute direct SQL tool can simplify your troubleshooting experience. You can download it using the following link - https://github.com/TrudAX/XppTools#installation

Feel free to post any comments(better as a GitHub issue) or ideas, what else can be improved.

A tool to convert a list of values to a query range

Wed, 08 Jul 2020 22:12:03 GMT

This functionality can help users deal with multiple values in a query range. You can:

Copy the list of values from Excel and paste it in a query range;
Easily manage the values in the existing range.

This tool was initially migrated to D365FO by Anton Kazarnovsky. Then it was slightly polished.

How to work with this tool

To use this feature, you need to go to Options-Page options and press Advanced filter or sort button(or use Ctrl + Shift + F3 shortcut).

A new button List of values was added on this form. If you press this button a new form will appear with two parts:

Column view (upper part);
Range friendly view (bottom part).

In the upper part (Column view) you can find a user-friendly view of the values in the selected range. Here you can:

Edit a list of values (it will be immediately reflected in Range friendly view);
Copy and insert a list of values from Excel;
Convert the value from the Range friendly view field to Column view by using Convert button.

In the bottom part (Range friendly view) you can find the resulting query range. Here you can:

Edit a list of values;
Apply text conversions:
- Add 'quotes' (“…”) at the start and at the end of each value;
- Add 'asterisk' (*…*) at the start and at the end of each value;
- Replace '_' and '.' symbols by the question mark ("?") in each value;
- Remove spaces (' ') from each value.

After you have finished editing of the values list, you can review the final range value in the bottom part, and press 'OK'. Then the query range will be updated with the values. Several new ranges for the same field will be created if the length of the value list doesn't fit in one range.

The standard system allows you to paste the value list from Excel, but the resulting range is limited to 4K chars(range size). So if you filter more(on average it is 300-400 invoices for example), you range will be truncated. Paste lists from Excel into filter fields with the "is one of" operator

The approach used in this tool also has a limit - 2100 parameters. This is a technical limitation (https://cloudblogs.microsoft.com/dynamics365/no-audience/2010/05/10/binding-operation-failed-to-allocate-buffer-space/) (thanks @IevgenMir for pointing on that)

Summary

List of Values to Range tool can simplify your work with multiple query ranges. You can download it using the following link - https://github.com/TrudAX/XppTools#installation

Feel free to post any comments(better as a GitHub issue) or ideas, what else can be improved.

A simple way to implement a parallel batch processing in X++

Wed, 01 Jul 2020 22:12:03 GMT

One of the powerful features of Dynamics 365 Finance and Operations is a Batch framework. In this post, I explain how you can convert your existing batch job to multi-threaded to increase its performance.

Initial example

Let's consider the following operation - a user dialog that processes customer transactions and performs an action at the end.

The logic is quite simple - loop thought all specified customer transactions and call process() function. In our case, it will sleep for the specified number of milliseconds. After the processing of the transactions, it runs a final function - in our case, it is just an Infolog message.

public void run()
{
    //1. data preparation
    info(strFmt("%1 - Start operation", AifUtil::applyUserPreferredTimeZoneOffset(DateTimeUtil::utcNow())));

    //2. Query Processing
    this.runQueryProcessing();

    //3.final task
    info(strfmt("%2 - %1 record(s) processed", SysQuery::countTotal(queryRun),
 AifUtil::applyUserPreferredTimeZoneOffset(DateTimeUtil::utcNow())));
}
public void runQueryProcessing()
{
    while (queryRun.next())
    {
        CustTrans   custTrans = queryRun.get(tablenum(CustTrans));
        this.processRecord(custTrans);
    }
}
public void processRecord(CustTrans  _custTrans)
{
    //do some job using _custTrans and transDate
    sleep(taskSleepTimeMs);
}

Standard USMF demo company has 1700 customer transactions, so if we run this job with no filters in a user interface or in a batch and specify 200ms to process one transaction, it will take 340 seconds for the whole job.

https://usnconeboxax1aos.cloud.onebox.dynamics.com/?mi=SysClassRunner&cls=DEVTutorialBatchSingleThread&cmp=USMF

Solution design principles

It is quite obvious that we can optimize this by running the code in parallel threads. Possible options of how this works are described in AX perf blog - Batch Parallelism in AX, but every approach from that article has its pros and cons.

What should be considered while designing a simplified solution for this:

A change should be simple and require minimum modifications to the original class. We don't want to create new classes or new tables to support parallel execution.
We can't run a single batch thread per one transaction - it will create a lot of overhead for a batch framework, so the solution should allow specifying maximum batch threads.
Execution flow in batch mode or in user interface should be exactly the same, better to avoid operations that can be run only in a batch.
We should support the final task, and it should be executed only once after transactions processing.

In most cases, we(as developers) should know how to split the load. In the example above we can just split selected customer transactions by equal intervals, but the split function can be more complex(for example we may want to avoid running parallel tasks for the same customer to prevent blocking)

The main idea is to introduce a new class parameter batchIdentifier - identification for the split interval and then run our logic only for this interval.

I created a base class DEVTutorialBatchMultipleThreadBase to incorporate this logic. If we have batchIdentifier specified - that means it is a child task that needs to perform a calculation for this batchIdentifier. If the class is executed with an empty batch identifier - that means it is the main task that should split the load and create all tasks for each split interval and the final task at the end.

class DEVTutorialBatchMultipleThreadBase extends RunBaseBatch
{
    public void run()
    {
        container               batchIdentifierCon;
        int                     i;
        ;
        if (batchIdentifier) //child task
        {
            if (batchIdentifier == this.finalTaskIdentifier())
            {
                this.runFinalTask();
            }
            else
            {
                this.runThreadTask();
            }
        }
        else
        {
            this.runStartTask();

            batchIdentifier = this.finalTaskIdentifier();
            this.processThreadItem(true); //create the final task, we need a dependency, so create it in the beggining.

            batchIdentifierCon = this.getBatchIdentifiersRangeCon();

            for (i = 1; i <= conLen(batchIdentifierCon); i++)
            {
                batchIdentifier = conPeek(batchIdentifierCon, i);
                this.processThreadItem(false);
            }

            if (finalTask)
            {
                if (this.isInBatch())
                {
                    batchHeader.save();
                }
                else
                {
                    finalTask.run();
                }
            }
        }
    }

Method processThreadItem creates the same instance of our class and calls a pack function. If the process is executed in a batch mode it creates a new runtime batch task. Without batch mode, it just runs this task.

Multiple threads batch example

Let's change our class to multithread.

we need to implement 3 function - runStartTask(), runThreadTask(), runFinalTask() to execute our tasks and getBatchIdentifiersRangeCon() to create a list of intervals - in our case it will be ranges FromRecId..ToRecId for the selected transactions. Method getBatchIdentifiersRangeCon() is the most complex and new in this example, all others are just a copy of original methods.

public class DEVTutorialBatchMultipleThread extends DEVTutorialBatchMultipleThreadBase
{
public void runStartTask()
{
    //1. data preparation
    info(strFmt("%1 - Start operation", AifUtil::applyUserPreferredTimeZoneOffset(DateTimeUtil::utcNow())));
}
public void runThreadTask()
{
    //2. Query Processing
    QueryBuildDataSource  qBDS = queryRun.query().dataSourceTable(tablenum(CustTrans));
    qBDS.addRange(fieldnum(CustTrans, RecId)).value(batchIdentifier);
    while (queryRun.next())
    {
        CustTrans   custTrans = queryRun.get(tablenum(CustTrans));
        this.processRecord(custTrans);
    }
}
public void runFinalTask()
{
    //3.final task
    info(strfmt("%2 - %1 record(s) processed", SysQuery::countTotal(queryRun),
                AifUtil::applyUserPreferredTimeZoneOffset(DateTimeUtil::utcNow())));
}
public container  getBatchIdentifiersRangeCon()
{
    container  res;
    QueryRun   queryRunLocal = new QueryRun(queryRun.query());

    QueryBuildDataSource   qBDS = queryRunLocal.query().dataSourceTable(tablenum(CustTrans));
    int                    totalRecords, curRecord, recordsPerBatch;
    RecId                  fromRecId, toRecId;

    qBDS.sortClear();
    qBDS.addSortField(fieldnum(CustTrans, RecId));
    !!!qBDS.addRange(fieldnum(CustTrans, RecId)).value(batchIdentifier);

    totalRecords = SysQuery::countTotal(queryRunLocal);
    recordsPerBatch = maxTaskCount > 0 ? totalRecords div maxTaskCount : totalRecords;       if (! recordsPerBatch)
    {
        recordsPerBatch = 1;
    }
    while (queryRunLocal.next())
    {
        CustTrans   custTrans = queryRunLocal.get(tablenum(CustTrans));
        if (! fromRecId) fromRecId = custTrans.RecId;

        curRecord++;
        toRecId = custTrans.RecId;
        if ((curRecord mod recordsPerBatch) == 0)
        {
            res += SysQuery::range(fromRecId, toRecId);
            fromRecId = 0;
            curRecord = 0;
        }
    }
    if (curRecord && fromRecId && toRecId) res += SysQuery::range(fromRecId, toRecId);
    return res;
}

In a user interface we added a new field "Number of batch tasks" to specify how many tasks to create.

if we run our function in a batch mode, we will see the following result

In this case, we got the total execution time of 30 sec, but values less than a minute are more related to the batch processing.

Summary

I described how easily you could implement multithreading in RunBase framework and convert the existing single-threaded task to the multi-threaded one. One note for this - use this approach only after you have performed all possible optimizations for the original code, running non-optimal code in multiple threads can create some problems. Also, the described solution is compatible with Ax2009 and AX2012 so that you can use exactly the same approach.

You can download all classes used in this post from the following links DEVTutorialBatchSingleThread, DEVTutorialBatchMultipleThreadBase, DEVTutorialBatchMultipleThread . If you find that something is missing or can be improved, don't hesitate to leave a comment.

Addin to generate Data provider and Controller class in X++

Wed, 17 Jun 2020 22:12:03 GMT

One of the most common applications of X++ SysOperation framework in D365FO is to build reports. In a common scenario, you need to create three classes: Contract, Data provider and Controller.

To generate all three classes, I extended my "TRUDUtils - DataContract class builder" Add-ins. You run it from the Dynamics 365 - Addins menu.

Let's consider the following example - a contract class AATestReportContract with 2 fields

Data provider

To generate a template code for a data provider class you need to specify "y" for "Generate report DP(y)" parameter and specify a data provider table(better to use TempDB table here).

The following class with 2 methods - getReportDataTmp() and processReport() will be generated

[SRSReportParameterAttribute(classStr(AATestReportContract)),
SRSReportQueryAttribute(queryStr(LedgerJournalTrans))]  //Change query
public class AATestReportDP extends SrsReportDataProviderPreProcessTempDB
{
    TmpLedgerConsTrans tmpLedgerConsTrans;

    [SRSReportDataSetAttribute(tableStr(TmpLedgerConsTrans))]
    public TmpLedgerConsTrans getReportDataTmp()
    {
        select * from tmpLedgerConsTrans;
        return tmpLedgerConsTrans;
    }

    public void processReport()
    {
        AATestReportContract   reportContract;
        Query           query;

        reportContract  = this.parmDataContract();
        query           = this.parmQuery();
        //populate tempdb table here..
    }
}

Controller

In order to generate a sample controller class, you need to specify "y" for "Generate report controller(y)" parameter.

In this case, the following class will be generated

public class AATestReportController extends SrsReportRunController
{
    protected void prePromptModifyContract()
    {
        AATestReportContract      contract;
        super();
        contract = this.parmReportContract().parmRdpContract() as AATestReportContract;
        //handle external record
        if (!args || ! args.record() || args.dataset() != tablenum(CustGroup))
        {
            throw error(strfmt("@GLS110030",tablestr(CustGroup)));
        }
        CustGroup custGroup = args.record();
        //contract.parmGroupId(custGroup.GroupId);
        //Query query = this.getFirstQuery();
    }

    public static void main (Args args)
    {
        AATestReportController         reportController;

        reportController  = new AATestReportController();

        reportController.parmArgs(args);
        reportController.parmReportName(ssrsReportStr(SalesInvoice,Report));
        reportController.parmShowDialog(true);
        reportController.startOperation();
    }
}

After generation, you need to modify a code according to your needs. Addin allows you to do less typing and focus more on report logic.

If you are new to SSRS reports development, Docentric has a great post Create a new custom SSRS report in D365FO. Also Microsoft Dynamics AX 2012 White Paper: Report Programming Model provides some development examples.

Summary

You can download this "DataContract class builder" from the following link https://github.com/TrudAX/TRUDUtilsD365/releases. If you find that something is missing or can be improved, don't hesitate to leave a comment.

How to create a new design for a Sales Invoice report in D365FO

Wed, 10 Jun 2020 20:12:03 GMT

Creating a new Sales Invoice report design using SSRS and X++ is quite a common task. In this post, I'll try to describe how you can do this in Dynamics 365 Finance and Operations.

Sales Invoice report development

I will use "DEVTutorial" prefix for new elements

Data provided class

Let's start with a data provider. We should create a new class and extend it from the standard SalesInvoiceDP. In this case, we can add additional fields that are relevant only for our report.

[SRSReportParameterAttribute(classStr(SalesInvoiceContract))]
class DEVTutorialSalesInvoiceDP  extends SalesInvoiceDP
{
protected void populateSalesInvoiceTmp(CustInvoiceJour _custInvoiceJour,
        CustInvoiceTrans _custInvoiceTrans,
        TaxSpec _taxSpec,
        CustPaymSchedLine _custPaymSchedLine,
        CustTrans _prepaymentCustTrans,
        TaxTrans _prepaymentTaxTrans)
    {
        super(_custInvoiceJour, _custInvoiceTrans, _taxSpec, _custPaymSchedLine, _prepaymentCustTrans, _prepaymentTaxTrans);
//populate additional fields in salesInvoiceTmp here
//salesInvoiceTmp.MyField = '';
//this method is called for main lines and also for all linked tables
//please note, that it will not be called if there are already some data exist in salesInvoiceTmp table for the selected Sales order, you need to delete existing records   }

protected void populateSalesInvoiceHeaderFooterTmp(CustInvoiceJour _custInvoiceJour, CompanyInfo _companyInfo)
{
    super(_custInvoiceJour, _companyInfo);
    //populate additional fields in salesInvoiceTmp here
    //salesInvoiceHeaderFooterTmp.MyField = '';
} }

Report design

Then add a new SSRS report to our project

We need to add Datasets for this report. In the standard system we have 4 sales invoice datasets, let's add them all.

Dataset name	Query
SalesInvoiceDS	SELECT * FROM DEVTutorialSalesInvoiceDP.SalesInvoiceTmp
SalesInvoiceHeaderFooterDS	SELECT * FROM DEVTutorialSalesInvoiceDP.SalesInvoiceHeaderFooterTmp
SalesInvoiceLocalizationDS	SELECT * FROM DEVTutorialSalesInvoiceDP.SalesInvoiceLocalizationTmp
DocuBrandDetailsDS	SELECT * FROM DEVTutorialSalesInvoiceDP.SysDocuBrandDetailsRegular

At the end you should have something like that:

Report Design

Then let's add a new Precision Design(Report)

Modify its properties: Size to standard A4 210x297mm and all Margins to 0.

Add the following sections

Header section

For the header, let's add a logo from Document branding

=First(Fields!Image1.Value, "DocuBrandDetailsDS")

and some client information

=First(Fields!SalesId.Value, "SalesInvoiceHeaderFooterDS")   & vbcrlf &  
First(Fields!InvoiceDate.Value, "SalesInvoiceHeaderFooterDS")  & vbcrlf &
First(Fields!InvoicingAddress.Value, "SalesInvoiceHeaderFooterDS")

Line section

Add a new Tablix control with SalesInvoiceDS dataset

Adjust Row visibility to the following formula "=IIF(Fields!ItemId.Value = "", true, false)" as SalesInvoiceDS may contain data from linked tables(like payment schedule)

Let's also add a totals section. The resulting design should look like this

Connect a new report to Print management

A sales invoice can be used in a Print management module. In order to connect our report to this module, we need to subscribe to two events(create 2 new classes):

Update 25/11/2021. A more detailed description can be found in the following blog How to Add New Report Format to Print Management

[ExtensionOf(classstr(PrintMgmtReportFormatPopulator))]
final class PrintMgmtReportFormatPopulatorDEVTutorial_Extension{   protected void addDocuments()
    {
        this.addStandard(PrintMgmtDocumentType::SalesOrderInvoice);
        next addDocuments();
    } }

final static class PrintMgmtDocTypeDEVTutorial_EventHandler{     [SubscribesTo(classstr(PrintMgmtDocType), delegatestr(PrintMgmtDocType, getDefaultReportFormatDelegate))]
    public static void getDefaultReportFormatDelegate(PrintMgmtDocumentType _docType, EventHandlerResult _result)
    {   switch (_docType)
        {
            case PrintMgmtDocumentType::SalesOrderInvoice:
                _result.result(ssrsReportStr(DEVTutorialSalesInvoice, Report));
                break;
        }  } }

Test Sales invoice report

That is all. It is time to test our report.

Build and deploy the report. Then, before the test, you need to add it to the print management settings. In order to do this - run the following job

class DEVTutorialPopulateReports
{   public static void main(Args _args)
    {
        PrintMgmtReportFormatPopulator::construct().populate();
    }}

https://usnconeboxax1aos.cloud.onebox.dynamics.com/?mi=SysClassRunner&cls=DEVTutorialPopulateReports

Next step is to add a logo to our report. Go to the Document images form and upload the image. Then go to the Branding details form and select this image.

To print the invoice go to the Accounts receivable - Invoice journal, select Document - Original preview and you should see our sample report:

The future of SSRS reporting

There may be a question of what is the future of SSRS reporting. Considering the latest news, I think that its popularity may grow. Microsoft recently released Power BI Paginated Reports that are completely based on SSRS "rdl" files. Technically a new Power BI Paginated Reports Builder is an old SSRS report builder with a new name(yes, another name change).

Check, for example, this recent course Power BI Paginated Reports in a Day course. Almost all its modules(except maybe 04) are relevant to D365FO development.

And this Bear in the previous picture is not just a funny picture, it is an official mascot of Power BI Paginated Reports. So with a new name and a new Bear, I hope the popularity of SSRS may grow.

SSRS reporting issues

If you watch this "Power BI Paginated reports Day course" you may feel that SSRS is cool(presentation is really great). However, the last main update for Report Builder was 10 years ago and the current version has some issues. Just some of them:

"rdl" file is a very complex XML format. So in most cases, you will not be able to compare changes in the report - by using text compare you will see a huge number of differences in XML structure even for a small design change.
Page splitting is very basic. For example, if you have a data row that consists of multiple lines and you want to avoid page breaks inside this row, you may find that this is not supported and you need to develop your own page grouping solution.
You can't define a section that prints something on the footer of the last report page(and it is a quite popular requirement for reports - often it contains some bank payment information). There is a workaround for this if your report has 1 or 2 pages - you can put this information on page footer and hide the element using Globals!PageNumber and Globals!TotalPages variables, but if a report has 3 or more pages this will leave an empty space on these pages.
Report designer doesn't support formulas for defining Width property.

So there are a lot of things that can be improved, and I hope Microsoft will invest in it.

Summary

Real-life invoice reports can be quite complex and require a lot of efforts to create them. However, as we see, creating a simple Sales Invoice design is not a complex task. You can use the project above as a starting point(another option to consider is to install modern design templates).

If you have some hints for SSRS report development, don't hesitate to post a comment. The project used in this post can be downloaded from my GitHub(you can download a model to a temporary folder and add elements from it to your project using "Add - Existing item.." command)

Understanding D365FO Version control system and why it is different from AX2012

Wed, 27 May 2020 22:12:03 GMT

If you are developing in Dynamics 365 Finance and Operations, you need to use Team Foundation Version Control(TFVC) system. It is not very complex, but its concept is different from what we have in AX2012.

In this post, I describe some scenarios that may produce unexpected results if you don't fully understand TFVC and use it in an "old AX2012 way".

On a typical AX2012 project, we usually have a common development application, where we are developing modifications, and when they are ready, we use import/export XPO file to transfer them on TEST application. Let's discuss how this may work in D365FO.

Initial setup

Here is our project setup, I created two branches one for our internal development and testing(Trunk - Main) and one for user testing - TEST. TEST is mapped to some temporary folder to allow merging.

Case 1 - Why my code doesn't compile

let's do our first modification. Add a new EDT EDT_FieldA and a new field based on this type to CustGroup table

Commit our changes as ProjectA

Then let's do our second modification - A new field to the same CustGroup table based on a new EDT EDT_FieldB

and then commit it to our Main branch as ProjectB

These two modifications are logically not linked, so after the testing, we can decide to move only ProjectB to the user test environment. It is a typical situation in AX2012, where you just export the project to XPO from DEV and load it into TEST(during the import you compare XPO with the existing code on TEST and transfer only relevant changes)

Some people may think that in D365FO the system can transfer only the relevant changes automatically and compare is not needed.

A typical mistake is to use Selected changesets merge to do this. Let see how it works:

You run a Merge command and choose Selected changesets

then you select only ProjectB assuming that only changes from ProjectB will be included.

But the result will be not as expected - you will also transfer a new FieldA and it refers to EDT that doesn't exist in TEST branch. So you will get a compile error when you try to Build your TEST branch(and that is not the worst scenario)

Case 2 - Why my test system doesn't work

Let's consider another example - you are developing some complex modification(Project1), and a part of this modification is a new validation on the Sales table

Another developer has developed in the same branch a simple modification just to provide some default values for the SalesTable(Project2)

Project2 is simple to test and after testing you decided to move to it user testing - TEST branch. Again this is a typical valid scenario in AX2009 or AX2012, we often don't care about other people modifications, we just transfer XPO and compare the code during the import.

You run "selected changesets" transfer and choose only Project2.

As the result, you will get a file that contains a part or Project1.

This will probably create an unexpected new validation on TEST branch, and your users may even not be able to create Sales orders due to this validation.

Case 3 - Why my code is missing after the merge

Let's continue with the example above. Consider that it is your lucky day and you noticed that you added unwanted code from Project1 to the TEST version before it's being deployed to users.

You open the class in TEST branch and delete the code related to Project1 from it. It is a typical AX2012 operation, you don't need this code now.

Then after some time, you finally decide to transfer Project1 to TEST from Main. You can even run full branch merge for this.

After a successful merge, your code related to Project1 will not appear in TEST. You lost it!

Microsoft recommendations

All examples above may happen due to incorrect usage of branching strategy.

The basic advice from Microsoft is the following:

Start with the main only branching strategy, branch strategically and adopt other strategies to evolve into more complex strategies as needed.

To achieve this, you need to work in sprints(for example 1 release per week).

It is a high chance that sometimes you need an emergency fix (that can't wait for a week). In this case, you restore a new temporary branch from the last build, implement a fix in this temporary branch and merge changes back to Main before the next release(so it will be a Release isolation strategy).

Full list of strategies is available here https://docs.microsoft.com/en-us/azure/devops/repos/tfvc/branching-strategies-with-tfvc?view=azure-devops, but please note that none of the recommended strategies allows "Merge selected changesets", you should always work with a forward-only merge. So merging just changeset2 on this diagram may corrupt your branch

The main issue that TFVC doesn't prevent from executing the wrong commands and probably in 95% cases when you run "Merge selected changesets" it will run without any issues as modifications usually don't contain common elements. And this may create a wrong feeling that it will always work.

Summary

One of the key features of Team Foundation Version Control(TFVC) is that changeset is a whole object, not just changes.

If you use the old AX approach("Merge selected changesets" in random order) with D365FO Team Foundation Version Control(TFVC) system, you may face problems. You may get compile errors, transfer unwanted code and lost changes in code.

So choose your branching strategy before the development, agree on it with a Project manager and follow it. Also If you know any tools that can prevent the described cases feel free to post a comment.

Resolving performance issues caused by parameters sniffing in AX2012 and D365FO

Thu, 07 May 2020 22:12:03 GMT

Let's discuss how AX performance problems caused by "parameters sniffing" may be resolved when you have on-premise system and access to SQL Server and for Dynamics 365 Finance and Operations.

AX2009/AX2012 on-premise

One of our clients reported a problem that his users complained about a slow AX performance. There was not just one operation, the general complain was something like: "System is very slow today - Posting is slow and invoicing is slow". When I logged in to the SQL box, the first metric I saw was a high CPU utilization that was above normal for this client.

After executing a top SQL query, I saw the following result. The first query took more than ten times logical reads than the second one.

The query was quite simple - get the last cost price for the item

@P1 nvarchar(5),@P2 nvarchar(21),@P3 int,@P4 datetime,@P5 nvarchar(5),@P6 nvarchar(11))SELECT A.ITEMID,A.VERSIONID,A.PRICETYPE,A.INVENTDIMID,A.MARKUP,A.PRICEUNIT,A.PRICE,A.PRICECALCID,A.UNITID,A.PRICEALLOCATEMARKUP,A.PRICEQTY,A.STDCOSTTRANSDATE,A.STDCOSTVOUCHER,A.COSTINGTYPE,A.ACTIVATIONDATE,A.MCSPMFBATCHSIZE,A.CREATEDDATETIME,A.RECVERSION,A.RECID FROM INVENTITEMPRICE A WHERE ((A.DATAAREAID=@P1) AND (((A.ITEMID=@P2) AND (A.PRICETYPE=@P3)) AND (A.ACTIVATIONDATE>@P4))) AND EXISTS (SELECT 'x' FROM INVENTDIM B WHERE ((B.DATAAREAID=@P5) AND ((B.INVENTDIMID=A.INVENTDIMID) AND (B.INVENTSITEID=@P6)))) ORDER BY A.DATAAREAID,A.ACTIVATIONDATE,A.CREATEDDATETIME DESC

But when I clicked on "query_plan" column in the output, I saw the reason why it took so much time – the wrong plan that started with InventDim table.

Probably recently they created a new location, and this new location was the first parameter for this query and SQL Server based on the current statistics decided to start the plan with InventDim(that was probably OK for this new location with no data), the plan was cached and caused a performance problem for all other locations.

The correct plan should be the following - start the search by ItemId and then check InventDim(it can be obtained by running Display Estimated Execution plan for the query above)

To avoid such situations in the future, I created a custom plan with a "force order" hint

EXEC sp_create_plan_guide @name = N'[AX_InventItemPriceDimSite]', @stmt = N'SELECT A.ITEMID,A.VERSIONID,A.PRICETYPE,A.INVENTDIMID,A.MARKUP,A.PRICEUNIT,A.PRICE,A.PRICECALCID,A.UNITID,A.PRICEALLOCATEMARKUP,A.PRICEQTY,A.STDCOSTTRANSDATE,A.STDCOSTVOUCHER,A.COSTINGTYPE,A.ACTIVATIONDATE,A.MCSPMFBATCHSIZE,A.CREATEDDATETIME,A.RECVERSION,A.RECID FROM INVENTITEMPRICE A WHERE ((A.DATAAREAID=@P1) AND (((A.ITEMID=@P2) AND (A.PRICETYPE=@P3)) AND (A.ACTIVATIONDATE>@P4))) AND EXISTS (SELECT ''x'' FROM INVENTDIM B WHERE ((B.DATAAREAID=@P5) AND ((B.INVENTDIMID=A.INVENTDIMID) AND (B.INVENTSITEID=@P6)))) ORDER BY A.DATAAREAID,A.ACTIVATIONDATE,A.CREATEDDATETIME DESC',
@type = N'SQL',
@module_or_batch = null,
@params = N'@P1 nvarchar(5),@P2 nvarchar(21),@P3 int,@P4 datetime,@P5 nvarchar(5),@P6 nvarchar(11)',
@hints = N'OPTION(force order, loop join)'

After that, the system performance restored to normal, and the CPU load was considerably reduced.

So the troubleshooting was quite easy to perform, the problem was fixed, and this issue will not happen again. The complexity of these steps is that you need someone who understands Dynamics AX DB structure and has access to the SQL Server.

If you have just a SQL administrator who doesn't know AX Database structure, you can create a simple instruction to deal with such problems:

Run a top SQL query and copy results to Excel(you can copy all columns except the last one - "query_plan")
Click on the last column - "query_plan" for the first 3-5 rows
Save them to separate files with .sqlplan extension
Try to clear the SQL cache with DBCC FREEPROCCACHE command

Then this information may be passed to AX developer in order to fix the problem.

The wrong way to resolve such problems is to perform one of these actions:

Restart the system
Run reindexing
Run statistics update

These actions will clear the current cache, and there is a high chance that the problem will disappear, but it will not be resolved.

A note about Plan Guides: ideally you should keep number of custom Plan Guides to minimum(as they may become invalid if you, for example, modify the table structure). After you fixed the issue with the plan guide, try to think how this can be avoided by modifying the code(typical solutions - create/update indexes, use forceliterals hint in X++, modify the query..)

Some troubleshooting tips(update 18/02/2021)

1.You can view existing plans with their hints in Plan Guides section

2.In order to check what plan hint is applied for the query, press Show execution Plan XML.. in the plan view, and check the last value for "StmtCursor" tag.

3.You can use a handy tool MSSQLPlanToSpExecuteSql by Denis Sukhotin to convert the plan XLM to a normal SQL with the actual values instead of @P1. Then you can execute this query from SSMS with different parameter values and try to understand why the wrong plan was used

Dynamics 365 Finance and Operations - cloud version

In the above situation, we had access to the SQL box, let's discuss how this can work when we don't have direct access to the SQL Server - in the new cloud D365FO.

Microsoft proposes to use LCS for this. On the Environment monitoring page you have a "SQL Insights" tab where you can execute the predefined "List most expensive queries" command(full documentation is here)

And to check the actual plan you need to get a result of this command and paste it to "Get the SQL query plan for a given plan ID" and this will give you a .sqlplan file that you can open in a local SQL Management studio.

Then there is another command to fix the plan.

So in theory, we should have almost the same tools as we have with on-premise system. But the current problem is that every action in LCS is very slow. I tested these functions on several production environments and typical response time was about 5-10 minutes(you just see "Please wait" message). This time is huge, and the whole system is quite unusable due to this. Moreover, you can't run this in different windows, only one request in time is allowed.

From the practical experience queries that work with SQL plan cache are almost instant even for huge databases, so the current problem is somewhere in communication between LCS and Azure SQL.

Conclusion

If you use the on-premise system(AX2009, AX2012) and have access to SQL Server, make sure that your administrators know how to diagnose and solve parameters sniffing issues. It may happen even for an old system, for example, a case from this blog happened to a seven year old system.

If you are in the cloud - vote for the idea to fix LCS Insights response time

Any comments are welcome

D365FO Table browser extension

Wed, 18 Mar 2020 20:12:03 GMT

The idea of this post is based on Alex Meyer's post How to Continue to Use the Table Browser Functionality Within D365FO. I created a new X++ utility - Table browser extension. It contains the following changes:

As Alex suggested in his post, I created a new menu item that allows running the table browser for a specific table:

**https://usnconeboxax1aos.cloud.onebox.dynamics.com**/?mi=DEVSysTableBrowser&cmp=DAT&tableName=UserInfo

-Allow editing

The standard table browser doesn't allow editing in non-development environments. This is controlled via the following standard code:

I created an extension that overrides this behaviour. If a user belongs to the new role "DEV Editable table browser", the table browser editing function is not blocked.

As I know the only way to edit data directly in tables for non-development Dynamics 365 Finance and Operation environment is either become the master of Electronic reporting or create a support request. These options may take some time, so it is good to have an "emergency" tool like this.

Summary

You can download the Editable table browser tool using the following link - https://github.com/TrudAX/XppTools#installation. I hope you may find this tool useful, also feel free to comment if you see something is missing

Visual Studio extensions to simplify your D365FO debug experience

Thu, 27 Feb 2020 20:12:03 GMT

Visual Studio Marketplace contains lots of tools, and some of them can be used with D365 Finance and Operations. In this post, I want to describe some useful Visual Studio extensions to simplify your D365FO debug experience.

Debug Attach Manager

The first one is a free, open-source tool Debug Attach Manager by Viktar Karpach. To debug the running D365FO application you need to go to Debug > Attach to process > then select iisexpress.exe process and press Attach.

But with Debug Attach Manager it is much simpler - it adds a new button that allows to do this in one click.

Update: the similar functionality exists in ReAttach.

Visual Commander

Visual Commander is designed to provide lightweight scripting to Visual Studio IDE. It is more like macro VBA scripts in Excel, but for Visual Studio. You can use C# to write new commands and a free version allows you to create 10 commands.

For example, you can create a command that adds a breakpoint to Infolog.add() method

EnvDTE.Debugger debugger = (EnvDTE.Debugger)DTE.Debugger;  
debugger.Breakpoints.Add("", @"K:\AosService\PackagesLocalDirectory\bin\XppSource\ApplicationPlatform\AxClass_Info.xpp", 298);
//use C:\AOSService\PackagesLocalDirectory\Bin\XppSource\ApplicationPlatform\AxClass_Info.xpp for Local VMs

or create a command that opens your favourite object

string absolutePath = @"K:\AosService\PackagesLocalDirectory\ApplicationPlatform\ApplicationPlatform\AxClass\Info.xml";
DTE.ItemOperations.OpenFile(absolutePath, null);

These commands will be available in VCmd menu

Summary

I hope you may find these tools useful, also feel free to comment if you see something is missing

Note: there is a strange bug when downloading extensions using IE from VS Marketplace site. It autosaves downloaded file as ".zip", but to install the extension, you need to rename this file to ".vsix"

Examples of AX2012/ AX2009 performance problems

Wed, 08 Jan 2020 20:12:03 GMT

Introduction

In this post, I try to describe some performance-related issues that I faced on client projects(mostly versions AX2009 and AX2012R3) and how they were resolved.

It is more like a collection of examples for my previous post Dynamics AX performance audit. It also illustrates the initial concept described in this post: a performance analysis should be performed from the highest level and to continue by going deeper in details.

There was a set of problems where users complained about the "slow system" in general

Shared hardware usage for AX SQL server

This issue was quite an unusual one. The whole AX implementation was installed on VM cluster where every CPU core shared between several VMs. The number of VMs allocated to the same CPU core is called CPU allocation ratio, and it was 1 to 5. You probably can save your money on hardware and electricity costs using such configuration, but as a result, overall system performance was very slow, especially during the day. Microsoft Dynamics AX 2012 System Requirements document clearly says that dedicated hardware should be used.

How to detect: I don't think is possible on Windows level, the best option is to ask IT support. Leading cloud providers always provide you with dedicated hardware

Database mirroring setup

One client set up a database mirroring using “High safety without automatic failover” mode. In this mode, a process waits for transactions to commit on both servers. This setting caused massive IO delays(like one record inserts for several seconds), not supported by Microsoft and solution was to switch on "High-performance" mode for mirroring settings.

How to detect: Such issue is quite easy to identify using Wait statistics script. If you see that more than 80% waits are disk related, it is a time to check what the reason for this is.

Slow performance due to low SQL Server memory

Dynamics AX is an OLTP system; it should not read any data from the disk during routine daily operations and SQL Server should have enough memory to hold the active data. If you don't have it - the available memory should be increased

How to detect: Check TOP SQL queries during the day – if you have a lot of physical disk reads, it is time to increase SQL Server memory(queries, of course, should have proper indexes)

SQL Server issues

Missing indexes

That is a common problem. Usually, you use a missing index query to find such indexes, but I also added a new column to TOP SQL query - "Has 99%" that allows to quickly see your TOP SQL queries that have 99% missing index impact recommendation

Parameters sniffing

Parameter sniffing can be a huge problem, especially if you have separate teams responsible for SQL Server and AX. We had a case where a client complained about periodic slow ledger journal posting that stopped the work of a whole accounting department.

In that case, the reason was just one custom query affected by parameters sniffing(not the optimal plan was initially selected and cached). Before the performance audit client spent about a month with tons of e-mails between AX team, SQL team and Microsoft support trying to implement typical recommendations like statistics refresh, re-indexes with a different fill-factor and so on...

The issue has been quickly resolved by creating a new plan guide with OPTIMIZE FOR UNKNOWN hint.

How to detect: To start with you need to analyse "Get Top SQL" output and check queries from the top. Often, the first step is to compare the actual plan with the Unknown parameters plan. To do this:

Copy a query to a new SQL window and replace “(“ with “declare” and remove the last “)”
Press “Display estimated execution plan” button

Compare it with the plan from the "Get Top SQL" output; if they are different, you need to analyse why.

You need to know what logical reason for this query is, how data is distributed in your tables, how unique(or how selective) each condition in the select query is, why SQL Server generates two different plans and which plan is the correct one.

To view actual parameters for the plan in the cache you need to press "Show Execution Plan XML.." on the plan itself and scroll to the bottom. You can use a handy tool MSSQLPlanToSpExecuteSql by Denis Sukhotin to convert this XLM to a normal SQL with the actual values instead of @P1. Then you can execute this query from SSMS with different parameter values.

Some queries can be fixed only by changing the X++ code. For example consider the following code, that gets total quantity for the specified Item on a License plate.

select sum(AvailPhysical) from inventSum
    where inventSum.ItemId == @P1
join inventDim
    where inventDim.inventDimId == inventSum.InventDimId &&
          inventDim.LicensePlateId == @P2;

"License plate" is a very selective field, so for the normal parameters InventDim(LicensePlateIdIdx) index will be used. But what happens if we try to execute this statement with @P2="". If we had this plan in the cache, the system would scan all InventDim records with empty LicensePlateId(and this will be a huge number of records) or if it were the first run, it would start processing this select with ItemId field, that is not so selective. Both options are incorrect and will lead to problems. But from the logical point of view LicensePlateId="" doesn't make any sense, so instead of trying to fix this statement, we need to fix its parameters. In this case, a developer just forgot to add a check for an empty value, and the solution was to add this check and not to run this statement if @P2 is empty.

The main advice here is never use statistics update or re-indexing to solve such issues, in the origin blog post I have some links what the approach should be(like additional indexes, hints in AX, convert to forceliterals, create custom plan..)

AOS and Terminal servers issues

Waiting time for batch tasks

A client complained about slow integration. During the analysis, I found that for the integration they used a batch job that should be executed every minute, and sometimes it wasn't.

There is one old recommendation on the Internet that when you set up an AX batch server, you should allocate only several threads per CPU core. In this case batch server was set up with 32 threads and such low setting caused batch execution delays(as there were no free threads available). To identify such problems, I created a SQL query that compares batch job "Planned execution start time" with the "Actual start time". If you see some differences between these numbers, you have the same problem.

My advice here is to increase this "Maximum batch threads" parameter for AOS until you have CPU and Memory resources(modern server can handle hundreds of threads). It is interesting that in the new D365FO version Microsoft has added Batch priority-based scheduling(PU31), but this feature probably just hides the problem.

Huge delays during sales orders posing and printing

That was a quite interesting and complex case. A client complained about periodic delays(1-2 minutes) during sales orders posting. The issue couldn't be replicated(initially it looked like typical parameters sniffing or blocking issue). The problem was that they also had a lot of other performance issues(a lot of missing indexes, old SQL version, High CPU load on SQL server and so on...). The client used AX2009 and they outsourced three different companies to support Infrastructure, SQL and AX side.

When we fixed all performance issues, the system load became low, but they still sometimes complained about the original issue and we still couldn't replicate it(in all our tests small orders posted within several seconds). Setting up a SQL blocking alert and analysing TOP SQL also didn't show any problem. We ended up with a monitoring modification that logged different stages of sales invoice posing. The first finding was that posting time didn't depend on the number of sales order lines, next iterations showed that delay was caused by the "print" method. Finally, we found the reason for this delay - users used Windows2016 Citrix clients(AX2009 does not support that) and in our tests we used Windows2008R2 RDP to run AX client. Printer driver reconfiguration solved this issue.

Slow standard operations

A lot of clients experienced issues with operations already fixed by Microsoft.

Top 2 problems that almost all clients had:

Batch tables deadlocks - was fixed in CU13 KB3209851Continuous deadlocks in batch tables - Microsoft switched locks processing to application locks
Locks and deadlocks on InventSumDeltaDim - was fixed in KB4019571 Deadlocking on InventSumDeltaDim causes Sales order release batch to fail. The solution is quite simple - delete RecId index and replace it with TTSItemCheckDimIdx

The best way to start any performance investigation if users are complaining to individual operations is to search for the solution on the LCS "Issue search"(you can search by description or by AOT element name). For example for AX2012R3, there are 120 resolved performance-related fixes to different parts of the system(you can view the full list searching by "slow")

Also, it is helpful to have the latest standard version on AX2012 and use it as a reference(if you have such application you can also extract hotfixes from it as xpo)

With the latest version definition there is a trick - if you google "latest AX version" you will probably find this page - Overview of Microsoft Dynamics AX build numbers. The problem with this site that it is not updated anymore. Current AX2012R3 latest version(both for the Binary and application) is called "August 2019 release". It can be downloaded from LCS - AX2012 project - Updates - UPDATE INSTALLER FOR MICROSOFT DYNAMICS AX 2012 R3

Application issues

I saw these settings that affected performance:

Unused Financial dimension sets

Some customers initially created a lot of “Financial dimension sets” but used only several of them. For example, the setup below means that when the system creates a one ledger transaction – 16 different dimension set transactions will be created(they are used in Trial balance report)

To get what was used, I created the following query(it displays the number of unprocessed records per dimension set):

select count(*) as NumberOfRecords, DimensionHierarchy.NAME from DIMENSIONFOCUSUNPROCESSEDTRANSACTIONS join DimensionHierarchy on DIMENSIONFOCUSUNPROCESSEDTRANSACTIONS.FOCUSDIMENSIONHIERARCHY = DimensionHierarchy.RECID
group by DIMENSIONFOCUSUNPROCESSEDTRANSACTIONS.FOCUSDIMENSIONHIERARCHY, DimensionHierarchy.NAME

The solution was to delete unused sets

Pending workflow tasks

If you use workflow, you need to check that the number of tasks in a Pending status will not grow. You can check for such tasks in the Workflow history form and if there are some old and unused Pending tasks - delete them.

Summary

In this post, I tried to provide some examples of what can be found during the performance review. The original post that describes the review process is here, all scripts related to it are available on my GitHub. If you see that something is missing or want to share some performance-related stories, feel free to post a comment.

D365FO class extensions naming tool

Thu, 12 Dec 2019 22:12:03 GMT

In D365FO Microsoft introduced a new concept of extension classes, which is different from standard elements. For example, for a form, you have the form design and related X++ code as one object. But for a form extension you create a form design as one AOT object and the code related to individual form extension components as separate AOT objects(extension classes). And this creates a new problem for developers - how to name these classes.

Microsoft published some basic guidelines for naming conventions - Naming guidelines for extensions-Naming extension classes, but it seems these rules are not even used inside Microsoft teams.

The page provides some basic rules - technically extension classes are not new objects, so Suffix should be used instead of Prefix, and you need to create a distinction between objects with the same names(like form CustTable and table CustTable)

Some objects from the standard code follow these rules, but many of them don't. Let's check standard objects in 10.0.5 application:

[ExtensionOf(tableStr(InventTable))]
public final class AppTroubleshooting_InventTable_Extension

According to the guide above it should be InventTableAppTroubleshooting_Extension

[ExtensionOf(formstr(BudgetReservationCreate_PSN))]
public final class BudgetReservationCreatePSN_Extension

[ExtensionOf(formStr(RetailStatementsPart))]
public final class RetailStatementsPart_Extension

These two are against the rule - "Don't name the extension just _Extension."

[ExtensionOf(formStr(SalesEditLines))]
internal final class RevRecSalesEditLines_Extension

[ExtensionOf(formStr(LedgerJournalTable))]
internal final class RevRecLedgerJournalTableForm_Extension

Here "RevRec" is used as a prefix instead of a suffix, but what is more interesting that for the first element they used an additional "Form" suffix but didn't add it to the second one.

So organising a naming convention even within Microsoft teams is a quite hard task.

Why this is important

Currently, there is no combined view in Visual Studio to view the final code(that includes all applied extensions and event handlers), so proper naming convention helps you to quickly find objects and avoid code duplication(where the same logical function is implemented in two different classes).

It doesn't matter what rules you will follow, more critical is that within one project naming convention should be the same.

Solution to this problem - automatic extension class creation

First, you need to define the naming conventions for your project. Currently, 8 elements can have extension classes; you need to define naming rules for all these elements(I have seen several dev guides where only 2 or 3 types were specified).

Class
Table
DataEntityView
View
Form
FormDataField
FormDataSource
FormControl

In addition to these rules, you also need to define where to put your event handlers. Technically event handlers are static methods and can be created in any class, but you should not put these static methods to extension classes(details can be found in this article - https://daxmusings.codecrib.com/2018/06/accidental-code-extensions.html). I think the best approach to use the same names as for extensions, but with the different suffix _EventHandler, and for forms put all handlers into one class.

So totally you need to define 16 rules.

I created a setup form where you can define these naming rules and added a function to quickly specify an initial template (currently there are only two: with "_" and without).

You can adjust the default settings and save it to file

Next step is to exclude manual work when you create extension classes. I extended my "Create extension class" add-in to automatically generate a class name based on these rules(this form can be called for any object that allows extension classes)

As a result, the following class will be generated

[ExtensionOf(formstr(CustTable))]
final class CustTableForm_TST_Extension

Summary

You can download this Create class extension from the following link https://github.com/TrudAX/TRUDUtilsD365/releases. If you find that something is missing or can be improved don't hesitate to leave a comment. If you have your own rules you can also send them to me(or create a Pull request) and I will add them as an option for the default settings.

Comparing D365FO development VMs performance

Wed, 11 Sep 2019 20:12:03 GMT

D365FO version requires more resources to perform development. In this post, I try to compare 3 popular development configurations and describe the pros and cons for each.

I have chosen the following VMs: Local Hyper-V Image, Azure VM based on SSD disks, Azure VM based on HDD disks.

There are detail specifications of these VMs:

Name	Local	SSD	HDD
VM Type	Local Hyper-V Image	Standard D8s v3	Standard D12 v2
CPU	Core i7-8700 3.2GHz, 6 cores, 24 GiB memory	E5-2673 v4 2.30GHz, 8 vcpus, 32 GiB memory	E5-2673 v4 2.30GHz, 4 vcpus, 28 GiB memory
Storage	Samsung 970(more than 100k IOPS)	5 premium disks	16 HDD disks
Run cost	Box for run 3 VMs - around 1.5k$	0.75$ per hour	0.52$ per hour
Storage cost	0	~4$ per day	~0

Both 3 VMs have used the same D365FO version - 10.0 PU24. After installation I disabled the following services:

Microsoft Dynamics 365 Unified Operations: Batch Management Service
Microsoft Dynamics 365 Unified Operations: Data Import Export Framework Service
Management Reporter 2012 Process Service

Overall system performance tests

For the compile task I got the following results:

Compile is a CPU intensive task.

What is interesting here - SSD VM time is very close to HDD despite of the fact that SSD has 8 cores and HDD has 4 cores. The main reason for this - is that Dv3 series has CPUs with Hyperthreading enabled but Dv2 has CPUs with Hyperthreading disabled. Hyperthreading gives only a small boost, that is why it is very important when you plan you virtual infrastructure to keep in mind - when we say for example '8 core VM' - whether these are real cores or virtual ones.

Local VM has CPU with a higher clock speed and CPU itself has 6 physical cores - that explains more than 2 times difference in performance compared to the Azure-based machines.

Next test is a database synchronize:

Synchronize operation is IO and CPU-intensive task. Local VM has a more powerful IO system, and a more powerful CPU, that explains the better result. SSD VM performed much better compared to HDD in this test due to faster disks.

Daily task tests

In order to test performance for more frequent developer tasks I chose 2 tasks - time to hit breakpoint and time to display 'Hello world' message from the job. In AX2012 both these tasks have near-zero execution time, you don't need to wait.

Time to hit breakpoint test

Before the test I opened the AX main screen, to warm the system cache.

To prepare for this test I switched off 'Load symbols for items in the solution'. Then I opened AOT, searched for SalesTable form and added it into the new project. Marked the form as a startup object. After that opened the code and added a new breakpoint to the init() method.

Time in this test - is the time between I pressed Start and the time when the breakpoint was hit.

Two tests were performed to see how the cache changes this time

HDD VM was very slow in this test. You need to wait almost 5 minutes to see your first breakpoint! SSD and Local VMs first run was the same, but the second run Local VM was faster.

Hello world test

This test was performed straight after the breakpoint test, but I restarted Visual Studio. In this test I created a new project and added a Runnable class with the following code

info("Hello world");

On the second run, I changed this test to "Hello world2". On the Third run I didn't change the text, just run the same job.

I measured the time between pressing Start and the time when the message displayed in the browser.

Here are the results:

You need to wait almost a minute for the HDD to see the results of a simple job. For Local and SSD it is 70% faster. Local VM also executed the job faster if it was not changed - probably due to a high single-core CPU speed(but you still need to wait 7 second, it is not instant)

Conclusion

For D365FO typical development actions can take longer time comparing to AX2012 where most operations were near-instant. But, by choosing a proper hardware for Development VM you can compensate this time to some extent. There is no right answer what to choose, you need to consider your situation:

If you want maximum performance and can buy a new hardware(and now you can buy a PC that has 10-20% more speed than mine) - Local VM is the best choice, but this VM is hard to manage.
Azure Premium disks VM(SSD) is the best choice if you can spend a little bit more for the cloud. It costs 6$ more per day(10 hours) than HDD, but provides a noticeable performance boost. But you need to pay for the storage - if you use this VM only sometimes it can be expensive.
Azure standard disks(HDD) VM is the most commonly used configuration, but according to my tests, it is not the best choice for an active project. If you just do a simple math(typical consulting rate is 180$ per hour(3$ per minute), 5 minutes wait for the first breakpoint will cost you more than the daily price difference with SSD VM)

I also will try to repeat the same tests after the next D365FO release, it will be interesting to compare the progress in this area.

How to increase your AX2012R3 database synchronisation speed

Wed, 04 Sep 2019 22:12:03 GMT

Time to synchronise AX2012 database was always a pain point for production databases. Recently I found an interesting AxForum topic by Masel that is probably worth sharing.

If you want to execute database synchronisation faster, you need to create the following plan guide in the main AX database(not the _model):

DECLARE @stmt nvarchar(max);
DECLARE @params nvarchar(max);
EXEC sp_get_query_template
    N'select name, change_tracking_state_desc from sys.columns AS Cols inner join sys.fulltext_index_columns AS FTSCols inner join sys.fulltext_indexes as FTS on FTSCols.object_id = FTS.object_id on Cols.object_id = FTSCols.object_id where Cols.column_id = FTSCols.column_id and Cols.object_id = object_id(''SYSINETCSS'')',
    @stmt OUTPUT,
    @params OUTPUT;
select @stmt;
EXEC sp_create_plan_guide
    N'SyncTemplateGuide',
    @stmt,
    N'TEMPLATE',
    NULL,
    @params,
    N'OPTION(PARAMETERIZATION FORCED)';

I tested this script on 2 different systems(the slow and fast one):

AX2012R3 CU9, Database on HDD disks, CPU: E5-2630 2.3GHz
AX2012R3 Feb2019, Database on SDD disks, CPU: Core i7-8700 3.5GHz

In both cases it gave a considerable synchronisation speed boost(about 300% for the fast system). You can see the sync time before and after applying this fix below:

The technique of this optimization is quite unusual, by creating this plan_guide you disable SQL plan creation based on actual SQL statement parameters. It is in general very similar to X++ forceplaceholders command. AX database synchronisation process sends a lot of small requests, and excluding the plan creation step gives this performance effect.

Enjoy and thank Masel for sharing this.

How to read Excel and CSV files in D365FO using X++

Wed, 03 Jul 2019 20:12:03 GMT

Sometimes you need to write X++ code to read the data from Excel(xlsx) or CSV files in D365FO.

Typical scenario for this:

User interface operation, for example, a dialog with some parameters, where user can specify a file. Then this operations reads the file and performs some actions(in some cases standard Excel add-in or Data management module or Electronic reporting can perform the same task)
A batch job that reads files from the network share(Azure storage, see example) and processes them(as standard DMF doesn't support import in transaction, in case you have multi-line documents, you need to write custom code)

In this blog post, I provide an example of X++ classes that can be used to read XLSX and CSV files.

File reading engines

To read an Excel file I use EPPlus library(https://github.com/JanKallman/EPPlus). The big advantage of this library that it is already a part of D365FO installation, you don't need to add external references.

With CSV file it is more complex. Standard D365FO installation doesn't include libraries that can read CSV format(it is quite complex). Often people use TextIO class that can read simple delimiter separated files, but this class can't handle more complex scenarios(for example delimiter symbol is presented in the data, or "new line" symbol is in the data). To read such files I use Microsoft.VisualBasic.FileIO library. It contains the proper CSV format reader, so if a file can be opened in Excel, this library can read it. (Note: This library is a part of D365FO installation and exists in .NET Core if you concern about future, but if you don't want to add a reference for some reason, I also included comments in DEVFileReaderCSV how to switch it to TextIO)

Reader classes

As reading CSV and XLSX is very similar from the programming perspective I created one base class DEVFileReaderBase and two derived DEVFileReaderCSV and DEVFileReaderExcel. Reading often includes the following stages:

Open the file and read its content to the container, close the file
Read the header row (if your file contains headers)
Loop though rows
Get the cell value for the current row (this can be done by column name - if you have the header row, or by column index)
Try to convert the cell value to the required type(for CSV it will be always "string to type" conversion, for xlsx - convert from the type of cell to the required AX type)

//Sample code to read an Excel file
DEVFileReaderExcel   fileReader   = new DEVFileReaderExcel();
fileReader.openFile(memoryStreamFile);
fileReader.readHeaderRow();
while (fileReader.readNextRow())
{
    info(strFmt("row: %1", fileReader.getCurRow()));
    info(strFmt("%1, %2, %3",
                fileReader.getStringByName("Main account"),
                fileReader.getStringByName("BusinessUnit"),
                fileReader.getRealByName("Amount")
               ));
}
//Sample code to read a CSV file
DEVFileReaderCSV   fileReader   = new DEVFileReaderCSV();
fileReader.readCSVFile(memoryStreamFile);
fileReader.readHeaderRow();
while (fileReader.readNextRow())
{
    info(strFmt("row: %1", fileReader.getCurRow()));
    info(strFmt("%1, %2, %3",
                fileReader.getStringByName("Main account"),
                fileReader.getStringByName("BusinessUnit"),
                fileReader.getRealByName("Amount")
               ));
}

Both examples print out the file content to the infolog

Generate user dialog

To create a user dialog for the file import I extended my Create RunBase class utility(https://github.com/TrudAX/TRUDUtilsD365#runbase-class-builder)

Right now it accepts Excel or CSV value in the "Add file upload" parameter.

If you enter the following parameters into this utility

DEVReadFromFileExample2
Read From Excel


excel

LedgerJournalNameIdDaily*
TransDate*

it automatically generates the required code to read a file in a RunBase dialog

Performance testing

Let's test the performance. To perform a test I created an Excel file with 10k lines and 10 columns with different types(100k cells total)

The main code for this performance testing is the following(full example is available in the DEVReadFromFileExamplePerf class) :

Results:

File type	Time to read 10k lines(sec)
Excel	1.54
CSV	0.56

As you see, reading itself is quite fast, in most cases you spend more time to process this data.

More complex example - Journal creation

Let's consider more complex example - create a ledger journal based on the data from Excel file.

In this case we will use an Excel file with 3 columns(Main account, BusinessUnit, Amount) and a user dialog with a "Journal name" parameter.

To generate a dialog class we need to specify the following parameters in Create RunBase class utility:

DEVReadFromFileExampleCreateJournal
Create ledger journal from Excel


excel

LedgerJournalNameIdDaily*

Journal creation logic can be copied from my previous post - https://denistrunin.com/xpptools-createledgerjournal/, dimension processing is from the following post - https://denistrunin.com/xpptools-devfindim/

In our case this code is used for journal creation:

void createLedgerJournal()
{
LedgerJournalTable      ledgerJournalTable;
LedgerJournalEngine     ledgerJournalEngine;
ledgerJournalTrans      ledgerJournalTrans;

ttsbegin;
while (fileReader.readNextRow())
{
    if (!ledgerJournalTable.RecId)
    {
        ledgerJournalTable.clear();
        ledgerJournalTable.initValue();
        ledgerJournalTable.JournalName = ledgerJournalNameIdDaily;        ledgerJournalTable.initFromLedgerJournalName();
        ledgerJournalTable.JournalNum = JournalTableData::newTable(ledgerJournalTable).nextJournalId();
        ledgerJournalTable.Name = strFmt("Excel file, Date %1", DEV::systemdateget());
        ledgerJournalTable.insert();

        info(strFmt("Journal %1 created", ledgerJournalTable.JournalNum));

        ledgerJournalEngine = LedgerJournalEngine::construct(ledgerJournalTable.JournalType);
        ledgerJournalEngine.newJournalActive(ledgerJournalTable);
    }

ledgerJournalTrans.clear();
ledgerJournalTrans.initValue();
ledgerJournalEngine.initValue(ledgerJournalTrans);
ledgerJournalTrans.JournalNum           =   ledgerJournalTable.JournalNum;
ledgerJournalTrans.TransDate            =   DEV::systemdateget();
ledgerJournalTrans.AccountType          =   LedgerJournalACType::Ledger;
ledgerJournalTrans.modifiedField(fieldNum(LedgerJournalTrans, AccountType));

ledgerJournalTrans.LedgerDimension = LedgerDimensionFacade::serviceCreateLedgerDimension(
LedgerDefaultAccountHelper::getDefaultAccountFromMainAccountId(fileReader.getStringByName('Main account')),
DEVDimensionHelper::setValueToDefaultDimension(0, DEVDimensionHelper::BusinessUnit, fileReader.getStringByName('BusinessUnit')));
if (!ledgerJournalTrans.LedgerDimension)
{
    throw error("Missing or invalid ledger dimension for journal process");
}
ledgerJournalTrans.modifiedField(fieldNum(LedgerJournalTrans, LedgerDimension));
ledgerJournalEngine.accountModified(LedgerJournalTrans);

//amounts
ledgerJournalTrans.CurrencyCode         =   Ledger::accountingCurrency();
ledgerJournalEngine.currencyModified(LedgerJournalTrans);
ledgerJournalTrans.amountCur2DebCred(fileReader.getRealByName('Amount'));
//additional fields
ledgerJournalTrans.Approver           = HcmWorker::userId2Worker(curuserid());
ledgerJournalTrans.Approved           = NoYes::Yes;

DEV::validateWriteRecordCheck(ledgerJournalTrans);
ledgerJournalTrans.insert();
ledgerJournalEngine.write(ledgerJournalTrans);
}
ttscommit;
}

As the result a new journal will be created

The full code is available in DEVReadFromFileExampleCreateJournal class

Summary

The following classes are used in this post:

DEVCommon model:

Classes: DEVFileReaderBase, DEVFileReaderCSV and DEVFileReaderExcel - read from CSV and XLSX files

DEVTutorial model:

Class DEVReadFromFileExamplePerf - measure read from file performance

Class DEVReadFromFileExampleCreateJournal - sample code to create a ledger journal from Excel file

Excel files used in this post can be downloaded here

Comments are welcome.

Dynamics AX performance audit

Wed, 05 Jun 2019 20:12:03 GMT

Introduction

Are your Dynamics AX users complaining about slow system performance? You started a research, found many tips on what to do, and still don’t know where to start. In this post I will discuss how to perform Dynamics AX performance audit to resolve performance problems. I will cover the following areas:

Current hardware analysis
SQL Server settings and blocking monitoring
AOS settings
X++ code optimization

I will also outline common performance myths and mistakes based on real-life scenarios.

The main principle of the performance audit is to start from the highest level(that allows to identify the performance problem with the minimum effort) and to continue by going deeper in the analysis. The described technique was used for more than ten customers with a number of users from 50 to 500 and database size from 100GB to 1TB. For all these customers, basic steps were the same and helped to identify and solve the problems.

Current hardware analysis

The initial task to begin with is to compare the used hardware with the specified one in a project Technical design document. It will not be good if you see a server design specification like "8-core CPU". Such phrases in Technical design means that you don't have a proper technical design. "8-core CPU" can mean any performance level. Try to filter "8 core CPU" on the Amazon, and you will see a considerable price variation

For a CPU price defines the performance level, and there will be noticeable differences of a system working on 10$ CPU compared to 10K$ CPU.

One of the clients complained about slow overall system performance. While comparing recommended hardware with the actual one, I have found that instead of 4-core 3.5GHz CPU, they had used 20-core 2.0GHz CPU. The client's original intention was correct, they thought that more cores mean more performance, but in that case, 4 cores were more than enough, but these should be the fast cores.

Hardware recommendations

For Ax2012 use the following guidance as a baseline.

AOS and Terminal servers

For AOS and Terminal server - CPU with the single thread performance equal to Azure Dv3 series. Memory - about 300-500MB per user

These are the average values. The current Intel CPU models give you about 30-50% more speed than Azure Dv3.

If your current CPU speed is below these D3 values, upgrading your servers can give you some noticeable performance boost. https://www.cpubenchmark.net/ is a web site where you can check the CPU performance level(you need to check single thread performance and total performance). If you don't like comparing these numbers, search on Amazon or eBay your current CPU price.

SQL Server

The same principles apply to SQL Server CPU, but they are not so simple. SQL Server is licensed by cores. For the wrong selected CPU model, your SQL Server license can cost more than the actual hardware. Check this blog post that explains the process in details. It is old, but the basic idea is that in every Intel model line there is a 4-core CPU with the maximum performance-per-core value, the more cores you get, the less performance per core there will be, so you should not use CPUs with more cores than your system needs.

As AX is an OLTP system, current CPU power for a SQL Server should allow processing data from a certain amount of memory. The amount of this memory is either the maximum amount supported by the SQL Server standard edition(64GB - pre-2016 and 128GB - for SQL 2016) or the maximum "active data" in the database.

From the practical experience modern 2 * 4 cores CPU can easily handle 128GB memory. 2 * 8 cores can work with 512GB. If you need more memory, probably it's time to think about 4-sockets server.

Storage system

Nowadays we have HDD, SSD, and NVMe storage. The main difference is the number IOPS they can handle(very simple 1K+, 20k+ 200k+). So if you have some storage problem(the main indication for this is IO related SQL waits), you can just upgrade to the next level.

AOS settings

There are several blog posts that cover optimal settings for the AOS.

https://community.dynamics.com/365/financeandoperations/b/axsupport/archive/2014/09/05/ax-performance-troubleshooting-checklist-part-1b-application-and-aos-configuration

https://blogs.msdn.microsoft.com/axinthefield/dynamics-ax-performance-step/

SQL Server analysis

The first thing to check is the current AX database size per table.

Analyzing database size

You don't need an exact table size(that takes time to perform), but you can get the size using the saved statistics from the sp_space_used procedure(execution time is quite fast). This script provides you the size per table. Copy the result to Excel and sort by size. Sometimes you see the picture like this:

in this case, a lot of space consumed by some temporary/tracing data and can be deleted.

Check the number of closed records in InventSum. If most of the records are closed, removing them can considerably increase the performance.

For example, if you get the following results, you can drop closed records(the only problem here is that some reports, for example, "On hand by date" can use closed records to display historical data, check this before deleting).

Use this link to find all standard cleanup jobs - Cleanup routines in Dynamics 365 for Finance and Operations

Also, check some system statistics. Often, you need to know the following:

Transactions per day(for logistics companies it will be a number of sales lines or invent trans per day) and the difference between peek and regular days)
Active users per day, per hour
Batch jobs and their timings

Compare these numbers with the numbers from the technical design, very often clients exceed them but "forget" to upgrade the hardware.

SQL server settings

To check the current SQL server settings, you need just one script - sp_Blitz. it performs thousands of checks and displays summarized recommendations with explanation links.

Missing indexes

Missing indexes script provides an overview of indexes that are considered missed by the plan guide engine

Output contains the following columns:

Table name
Equality columns: ''='' condition in the WHERE clause
Inequality columns: some logical condition(<", "!=") in the WHERE clause

Don't just follow these recommendations, every recommendation should be analysed from a logical point of view, and only indexes that really limit the actual search should be created. You don't need to analyse the whole output, often 30-50 top recommendations are enough.

Sometimes you can see recommendations that were caused by wrong conditions(mandatory fields missed in the SQL statement). In this case it is better to find and correct the statement, rather than trying to create an index(in the example below RefCompanyId field is missing for DocuRef selection)

Unused indexes

The script provides some unused indexes statistics for the large tables.

Remove the indexes only if they are related to not used functionality or a different country. In some specific cases for large tables, you can also consider disabling Partition and DataArea fields.

Wait statistics

Script gives you an overview of current wait events. If you see some IO related events here, analyze disk activity by disk and by file.

Top SQL analysis

The script provides active Top SQL commands, their statistics and saved plan from the SQL server statistics(to clear the list use DBCC FREEPROCCACHE command).

Ideally, you should know first 5-10 statements from this list and analyse the following:

They logically make sense for the current implementation - the number of executions and the statement itself relate to the current system functions. Very often I saw some statements here caused by incorrect system setup, by not used functionality or they are not related to AX at all.
They use optimal plans and indexes

FETCH_API_CURSOR here means some query from an AX form. To find the original query use Cursors for the session script.

AX long SQL statements tracing

You can enable tracing of long SQL statements in the user options. Use this job to enable it for all users. Often you need to analyse statements with the execution time more than 2-5 seconds.

A great advantage of this in AX is that you can see a stack trace of the statement. Also, keep in mind that a long execution time can be caused by 2 reasons:

A statement is heavy and it takes a long time to execute it
A statement is blocked by another session

Blocking analysis

The following reasons can cause unwanted blocking:

Group update operation(like update_recordset) in X++. Check this article that explains the problem in detail. To resolve this, you need either replace "update_recordset" with "while select forupdate" or adjust indexes
Blocking escalation - if you modify more than 5000 records in one transaction sometimes, SQL Server decides to escalate the blocking level. If you have a lot of memory, you can disable this behaviour(ALTER TABLE myTable set (LOCK_ESCALATION = DISABLE)), but first, check that you really need to update all records in one transaction.

A great instrument to deal with the blocking is to enable AX long SQL statements tracing(see above) - in this case, you will see the statement, user and actual operation(by using X++ stack trace).

It is also useful to enable context_info for the SQL session. This allows you to link AX session with the SQL Server SPID and find a blocked user.

The most challenging part of resolving blocking problems is to find an operation that causes blocking. The primary technique is to try to search blocking on the TEST version. In this case, you run the client, execute the operation(for example, post the sales order) and put a breakpoint to the last ttscommit statement for this operation. Then run another client and start executing operations(like another sales order or journal posting). If you catch a SQL block, you can easily implement and test a fix for it.

Parameters sniffing

Parameters sniffing quite often becomes a reason for performance problems. For example, you have a warehouse where most of the items have one batch ID, but there are some Items and Batches with generic names (like “No batch”, “Empty”). In this situation, when you query item by batch, your actual plan depends on the first query values. If your query contains “generic” batch values, SQL Server creates a plan, that starts execution with the ItemId, as a “generic batch value” is not selective in this case. But for most of your items, it will be wrong and gives a huge overhead, as in most cases Batch is a unique value.

There is no universal way to resolve Parameters sniffing issues(refer to the excellent BrentOzar post that describes this https://www.brentozar.com/archive/2013/06/the-elephant-and-the-mouse-or-parameter-sniffing-in-sql-server/ ), but there are several ways to deal with it in AX:

Create new indexes – you can create new indexes, to help SQL server always choose the best plan. This often works only if you have conditions to one table only
Use the sp_create_plan_guide command to force the actual plan – using this option creates a big admin overhead as you need to maintain these created plans. If you add a new field to the AX table you will need to change all the plan guides in which this table is used. Often you need to provide just a OPTIMIZE FOR UNKNOWN hint to disable sniffing for the SQL statement, better do not specify specific indexes
forceLiterals X++ hint –you send actual values to the SQL Server, and it chooses the best plan for these values. The overhead is that the plan will need to be compiled every time
Use index hint (new feature D365FO only)

You can view created plan guides in the Programmability section:

As a basic rule add forceliterals hint(or query.literals(true)) for the single SQL statements and create a plan guide(with the OPTIMIZE FOR UNKNOWN hint) for small SQL statements. Forceliterals usage can slow down your server. Check this article for the details.

Determining which statements are affected by parameters sniffing also can be tricky. Often you analyse statements from the Top SQL output(see above), then compare the actual plan with the estimated plan and if they are different, check the XML representation of the actual plan. At the end of this XML you can find initial(sniffed) parameter values and basing on them decide is that an issue and how it should be fixed.

Index rebuild and statistics update

Rebuild index and statistics update jobs can cause a lot of problems.

The main misconception is that these operations can improve system performance but in reality they can't. The side effect of these operations is that they clear the plan cache(the same can be done just with DBCC FREEPROCCACHE command) and due to parameters sniffing this is often considered as an improvement.

The typical scenario of going into this trap(Brent call it Index Maintenance Madness, but in my experience, the same relates to statistics update):

System suddenly becomes very slow(due to parameters sniffing issue)
Someone decides to run an index rebuild or/and a statistics update job
This magically helps and these operations started to be executed daily(but you can't resolve parameters sniffing issue with this)
Then system slowness happens during the day and statistics update also helps.
Frequency changes to 2 times a day, then 3 times a day and so on...

At the end of these events, a customer decides that index rebuild or/and statistics update job should run constantly and the server load they produce can even exceed the AX business logic load, cause blocking and slow down the system.

Check this article(Index Maintenance Madness and a great video from Brent Ozar that explains the theory and psychology in details.

Recommendations here is that you should not execute these operations in any way they can affect system performance. If you have a free maintenance window, run them once a week using Ola Hallengren‘s IndexOptimize procedure. It is more efficient than the standard SQL agent tasks as they allow you to execute UPDATE STATISTICS command without "WITH PERCENTAGE" clause and for the index rebuild you can specify fragmentation limits. Check this article for the details - Managing Index Fragmentation.

Don't use rebuild index and statistics update jobs to resolve any performance problems.

Code optimization

Before individual operations optimization(x++ code) you need to know the following:

Detailed problem description
Reproduction steps. That is the most complicated part, sometimes reproducing the problem can take more time than the optimization
What is the current and what is the required execution time
How you can test the optimization

Testing is quite easy for reports or operations that don't modify the data, but can be complex for posting operations. Try to minimize the number of manual testing steps, ideally, the test should be possible to run by one click(in this case you can try different ideas during the optimization process). For posting often you write a job that opens a transaction at the beginning and then generate a ttsabort at the end.

Trace Parser is a great tool for analysing operations execution. It will trace all sessions in the AOS and provides detailed output. For production tracing better to have a dedicated AOS as trace file can grow very rapidly when you have a lot of user sessions.

The advice here is to install and setup it before the actual problem happens. In this case, you can easily run the trace when needed.

Summary

Using these basic steps you can resolve your Dynamics AX performance problems and make your users happy. All scripts related to this post are available on my GitHub. I also described some performance-related issues that were resolved using the provided approach - Examples of AX2012/ AX2009 performance problems.

Update 10/10/2020 - If you want to know how to organize a performance optimization project, check this post.

If you need help you can contact me. Or if you see that some useful staff is missing, feel free to post a comment.

There are some Microsoft blog posts about performance: Managing general performance issues in Microsoft Dynamics AX, Analysis scripts for Performance Analyzer v2.0.

Also worth to check this great SQL Server resources collection - SQL Server KIT.

How to design a successful integration

Wed, 17 Apr 2019 22:12:03 GMT

There are many ways to implement integration between D365FO and external systems. To choose the correct approach you need to clearly understand integration requirements before doing actual programming/setup. In this document, I propose the actual checklist of what you need to consider to successfully design an integration solution. The checklist is based on Oracle guidelines and contains questions that you need to ask while designing integration and before choosing between different integration methods(https://github.com/TrudAX/TRUDScripts/blob/master/Documents/Integration/Integration%20Data%20Flow%20Requirements.md):

Defining the Integration Solution Topology
Defining Data Flow Requirements
Analysing the Data Flow properties
Defining the Quality of Service
Performance
Availability and Reliability
Security
Scalability
Logging and Nonrepudiation

Every question in this document is important and can dramatically change the integration approach. For example, if you have an external system that needs to query D365FO for some information(like item prices) and it requires 24x7 availability, you can't use any direct calls(like OData, Webservices) to D365FO as it obviously will not be available 24x7(during new releases, hotfix installations..)

Keys to the successful integration

Integration has never been easy. Here I try to describe the most problematic moments.

Performance

You need to clearly define and test the amount of data that needs to be processed and the time interval when processing should be done.

Even simple solutions can cause problems. For example, the client can ask you to develop a webservice that returns a list of vendors. It can be easily developed and tested without asking additional questions, but on production system you may see hundreds of calls per minute to this webservice that may completely stop the whole system and make your client unhappy. By asking a simple question from this checklist - "Why do you need this webservice and how often do you plan to call it? and getting the answer - "One call per document line" you can fix the proposed design before it is implemented.

Transaction support

When working with business documents transaction support is often required. For example, you import 10-line sales order from the external system, and 1 line can't be created due to some validation. In most cases you don't want to create a sales order with only 9 lines, the whole document should be rejected and often this behavior is implemented via SQL transaction support. This requirement may limit possible solutions as standard D365FO Data management module doesn't support transactions, you can't write several records in one transaction

In this case, you either need to implement your own transaction system(via additional flags) or write more custom code that was initially planned

Logging and traceability

Logging and traceability should be a base requirement. For export scenarios, it should be easy to identify what was exported and when this particular record was processed, for import scenarios - what the original incoming request, its processing status and created documents were. And vice verse - by the document you should be able to identify the original integration message.

Errors handling

Errors are often divided into 2 categories - that can and that can' be resolved by subsequent executions. For example, when you are reading data from the file, and its structure is not what you expect to see - you can mark this file with an error flag and move it to the Errors folder. If the file contains a vendor code that doesn't exist in the system, probably you can just show an error and then try to process this file again(a vendor may be created later).

You should also think if there will not be any reaction from the system administrator to these errors. Growing numbers of old errors can stop new messages from processing, so you should implement some logic to mark such messages as skipped after some time(or a number of processing attempts). Import and processing stage separation(via the staging tables) is also a good idea.

File usage

As D365FO is a cloud-based system often people ask: what about our existing file based integration that reads/writes files from the network share? And the answer is: if you want you can use the same approach in the cloud by replacing network storage to Azure file storage. From the user perspective it can look the same as a mapped drive in Windows. Libraries to work with the Azure storage are very similar to the File libraries(see the example here and one of the options to store connection strings here )

Async and sync

From my experience, it is better to try to avoid synchronous calls, especially when you don't control the external system. It's is not always possible, but if you can do this, better to implement some middleware storage where you can read/write messages. So if the 3-party app doesn't need real-time access, instead of providing ODATA or Webservice interfaces better to upload the data they required to SQL Azure database, File storage or somewhere else. It can solve a lot of problems - they can't slow down your system by querying a lot of data and you avoid complains like "while making ODATA request we get a timeout error".

Reproduction and testing support

It should be easy to reproduce/test single message processing. If your integration works with physical devices, you should think about a device emulator; if you are processing files from the specified folder(or processing data from the messages table), your solution should allow to select an individual file/message to process. This allows to quickly debug, test and solve potentials problems.

Summary

I uploaded this checklist to GitHub. If you see that some questions are missing, feel free to post a comment.

Create ledger journals in D365FO using X++

Tue, 16 Apr 2019 20:12:03 GMT

Creating LedgerJournalTrans using X++ is quite a common task, but sometimes I still see some mistakes(mostly related to fields initialization from account/offset account and voucher assignment). In this post, I'll try to describe possible options how to create ledger journals in D365FO.

Test scenario

To test different methods, I wrote a Runnable class DEVTutorialCreateLedgerJournal that creates a new journal based on the data from the existing source journal using different methods. The dialog looks like this

Let's discuss possible copy options(Copy type parameter):

LedgerJournalEngine class

This method is using LedgerJournalEngine class - the same class that is used when the user creates a journal manually on the Journal lines form. This increases the chances that the resulting line will be the same as a manually created line.

while select ledgerJournalTransOrig
    order by RecId
            where ledgerJournalTransOrig.JournalNum == _ledgerJournalTableOrig.JournalNum
{
    numLines++;
    if (!ledgerJournalTable.RecId)
    {
        //Often journal name from parameters is specified here
        DEV::validateCursorField(ledgerJournalName, fieldNum(LedgerJournalName, JournalName));

        ledgerJournalTable.clear();
        ledgerJournalTable.initValue();
        ledgerJournalTable.JournalName = ledgerJournalName.JournalName;
        ledgerJournalTable.initFromLedgerJournalName();
        ledgerJournalTable.JournalNum = JournalTableData::newTable(ledgerJournalTable).nextJournalId();
        ledgerJournalTable.Name = strFmt("Copy of %1, Date %2", _ledgerJournalTableOrig.JournalNum, DEV::systemdateget());
        ledgerJournalTable.insert();

        info(strFmt("Journal %1 created", ledgerJournalTable.JournalNum));

        ledgerJournalEngine = LedgerJournalEngine::construct(ledgerJournalTable.JournalType);
        ledgerJournalEngine.newJournalActive(ledgerJournalTable);    }

    ledgerJournalTrans.clear();
    ledgerJournalTrans.initValue();
    ledgerJournalEngine.initValue(ledgerJournalTrans);
    ledgerJournalTrans.JournalNum           =   ledgerJournalTable.JournalNum;
    ledgerJournalTrans.TransDate            =   DEV::systemdateget();
    ledgerJournalTrans.AccountType          =   ledgerJournalTransOrig.AccountType;
    ledgerJournalTrans.modifiedField(fieldNum(LedgerJournalTrans, AccountType));

    ledgerJournalTrans.LedgerDimension = ledgerJournalTransOrig.LedgerDimension;
    if (!ledgerJournalTrans.LedgerDimension)
    {
        throw error("Missing or invalid ledger dimension for journal process");
    }
    ledgerJournalTrans.modifiedField(fieldNum(LedgerJournalTrans, LedgerDimension));
    ledgerJournalEngine.accountModified(LedgerJournalTrans);
    ledgerJournalTrans.OffsetAccountType = ledgerJournalTransOrig.OffsetAccountType;
    ledgerJournalTrans.modifiedField(fieldNum(LedgerJournalTrans, OffsetAccountType));
    ledgerJournalTrans.OffsetLedgerDimension = ledgerJournalTransOrig.OffsetLedgerDimension;
    ledgerJournalTrans.modifiedField(fieldNum(LedgerJournalTrans, OffsetLedgerDimension));
    ledgerJournalEngine.offsetAccountModified(ledgerJournalTrans);

    //amounts
    LedgerJournalTrans.CurrencyCode         =   ledgerJournalTransOrig.CurrencyCode;
    ledgerJournalEngine.currencyModified(LedgerJournalTrans);
    LedgerJournalTrans.AmountCurCredit      =   ledgerJournalTransOrig.AmountCurCredit;
    LedgerJournalTrans.AmountCurDebit       =   ledgerJournalTransOrig.AmountCurDebit;

    //additional fields
    LedgerJournalTrans.Approver           = HcmWorker::userId2Worker(curuserid());
    LedgerJournalTrans.Approved           = NoYes::Yes;
    ledgerJournalTrans.Txt                = ledgerJournalTransOrig.Txt;
    LedgerJournalTrans.SkipBlockedForManualEntryCheck = true;

    DEV::validateWriteRecordCheck(ledgerJournalTrans);
    ledgerJournalTrans.insert();
    ledgerJournalEngine.write(ledgerJournalTrans);}

When you modify Account/Offset account fields in this example you need to call two methods(on ledgerJournalTrans and ledgerJournalEngine). This ensures that the line will be properly initialized from the Account/Offset account field.

ledgerJournalTrans.modifiedField(fieldNum(LedgerJournalTrans, LedgerDimension));  
ledgerJournalEngine.accountModified(LedgerJournalTrans);

Voucher assignment here is processing in ledgerJournalEngine.write().

Also, an interesting flag here is LedgerJournalTrans.SkipBlockedForManualEntryCheck. It is useful when you don't want to allow users to create manually lines with the same accounts as in your procedure.

LedgerJournalTrans defaultRow() method

This is a new approach in D365FO and it uses a new defaultRow() table method. Data entities also call this method during the import process. Its idea is that we don't control the sequence of different modifiedField method calls, we just populate the fields that we know, all other logic happens in the defaultRow() method.

Code for journal creation in this case:

while select ledgerJournalTransOrig
    order by RecId
    where ledgerJournalTransOrig.JournalNum == _ledgerJournalTableOrig.JournalNum
{
    numLines++;

    if (!ledgerJournalTable.RecId)
    {
        ledgerJournalTable.clear();
        ledgerJournalTable.initValue();
        ledgerJournalTable.JournalName = _ledgerJournalTableOrig.JournalName;
        ledgerJournalTable.initFromLedgerJournalName();
        ledgerJournalTable.JournalNum = JournalTableData::newTable(ledgerJournalTable).nextJournalId();
        ledgerJournalTable.Name = strFmt("Copy of %1, Date %2", _ledgerJournalTableOrig.JournalNum, DEV::systemdateget());
        ledgerJournalTable.insert();

        info(strFmt("Journal %1 created", ledgerJournalTable.JournalNum));
    }

    ledgerJournalTrans.clear();
    ledgerJournalTrans.initValue();

    ledgerJournalTrans.JournalNum             = ledgerJournalTable.JournalNum;
    ledgerJournalTrans.TransDate              = DEV::systemdateget();
    ledgerJournalTrans.AccountType            = ledgerJournalTransOrig.AccountType;
    ledgerJournalTrans.LedgerDimension        = ledgerJournalTransOrig.LedgerDimension;
    ledgerJournalTrans.DefaultDimension       = ledgerJournalTransOrig.DefaultDimension;

    ledgerJournalTrans.OffsetAccountType      = ledgerJournalTransOrig.OffsetAccountType;
    ledgerJournalTrans.OffsetLedgerDimension  = ledgerJournalTransOrig.OffsetLedgerDimension;
    ledgerJournalTrans.OffsetDefaultDimension = ledgerJournalTransOrig.OffsetDefaultDimension;

    ledgerJournalTrans.CurrencyCode           =   ledgerJournalTransOrig.CurrencyCode;
    ledgerJournalTrans.AmountCurCredit        =   ledgerJournalTransOrig.AmountCurCredit;
    ledgerJournalTrans.AmountCurDebit         =   ledgerJournalTransOrig.AmountCurDebit;

    //addition fields
    ledgerJournalTrans.Approver           = HcmWorker::userId2Worker(curuserid());
    ledgerJournalTrans.Approved           = NoYes::Yes;
    ledgerJournalTrans.Txt                = ledgerJournalTransOrig.Txt;
    ledgerJournalTrans.SkipBlockedForManualEntryCheck = true;

    ledgerJournalTrans.defaultRow();
    DEV::validateWriteRecordCheck(ledgerJournalTrans);
    ledgerJournalTrans.insert();
}

You can find an example of this approach in the MCRLedgerJournal class (and the usage is here)

Data entity

We can also use a data entity for journal creation. Technically it produces the same result as the previous method because a data entity at the end will call a table defaultRow() method.

while select ledgerJournalTransOrig
    order by RecId
    where ledgerJournalTransOrig.JournalNum == _ledgerJournalTableOrig.JournalNum
{
    numLines++;
    select dimensionCombinationEntity
        where dimensionCombinationEntity.RecId == ledgerJournalTransOrig.LedgerDimension;
    select dimensionCombinationEntityOffset
        where dimensionCombinationEntityOffset.RecId == ledgerJournalTransOrig.OffsetLedgerDimension;

    ledgerJournalEntity.initValue();
    ledgerJournalEntity.JournalBatchNumber     = journalNum;
    ledgerJournalEntity.Description            = strFmt("Copy of %1, Date %2", _ledgerJournalTableOrig.JournalNum, DEV::systemdateget());
    ledgerJournalEntity.JournalName            = _ledgerJournalTableOrig.JournalName;
    ledgerJournalEntity.LineNumber++;

    //if you have string and want to convert to ID - DimensionDefaultResolver::newResolver(_dimensionDisplayValue).resolve();
    //AX > General ledge > Chart of accounts > Dimensions > Financial dimension configuration for integrating applications
    ledgerJournalEntity.AccountType                 = ledgerJournalTransOrig.AccountType;
    ledgerJournalEntity.AccountDisplayValue         = dimensionCombinationEntity.DisplayValue;

    ledgerJournalEntity.OffsetAccountType           = ledgerJournalTransOrig.OffsetAccountType;
    ledgerJournalEntity.OffsetAccountDisplayValue   = dimensionCombinationEntityOffset.DisplayValue;

    ledgerJournalEntity.CreditAmount                = ledgerJournalTransOrig.AmountCurCredit;
    ledgerJournalEntity.DebitAmount                 = ledgerJournalTransOrig.AmountCurDebit;
    ledgerJournalEntity.CurrencyCode                = ledgerJournalTransOrig.CurrencyCode;
    ledgerJournalEntity.TEXT                        = ledgerJournalTransOrig.Txt;
    ledgerJournalEntity.TRANSDATE                   = ledgerJournalTransOrig.TransDate;

    ledgerJournalEntity.defaultRow();
    DEV::validateWriteRecordCheck(ledgerJournalEntity);
    ledgerJournalEntity.insert();

    journalNum = ledgerJournalEntity.JournalBatchNumber;
}

As you see this method requires less code: we don't even need to write code for the journal header creation, it is all handled by the data entity insert() method. Dimensions can be also specified as strings. However there are some limitations: sometimes data entity doesn't contain all the table fields and it doesn't support all account types.

Performance testing

Let's test the performance. First I created a test journal with the 1000 lines(**createByCombination method) and then copied it using these 3 different methods. I got the following results:

Method	Time to create 1000 lines(sec)
Using ledgerJournalEngine	30.54
Using DataEntity	33.09
Using Table defaultRow method	15.18

There are some differences between the copy speed in my example, but it is caused by a different logic for the dimension creation, so the result is that all methods are almost equal and quite fast. In a real-life scenario, you can expect an insert speed 10-30 lines per second.

Choosing the right method and things to avoid

In general, you have 2 options - to create a journal similar to the manual user entry or create it similar to the import procedure(for the second scenario choice between entity and table mostly depends on what input data you have and whether the entity supports all the required fields). So the choice between these two should be made by answering the question: if the user wants to create the same journal manually, does he use manual entry or data import? Use createJournalUsingLedgerJournalEngine as default, as it is more flexible.

Probably in D365FO it is better to avoid creation using JournalTransData classes or when you simply populate ledgerJournalTrans fields and call insert(). This initially can work, but later users may complain - e.g. "Why when I create a journal manually and specify a vendor account the Due date field is calculated, but your procedure doesn't fill it".

Summary

You can download this class using the following link https://github.com/TrudAX/XppTools/blob/master/DEVTutorial/DEVTutorial/AxClass/DEVTutorialCreateLedgerJournal.xml. The idea is that you can use this code as a template when you have a task to create(or post) a ledger journal using X++. Comments are welcome.

Tool to filter in a query as a developer

Thu, 07 Mar 2019 22:12:03 GMT

In the standard Advanced query filter dialog, when you select a field to filter, standard lookup displays only a field label. Sometimes, it is more convenient to filter by field AOT name(and not by the label).

This tool adds a new column in the standard Field lookup dialog, so you can search the required field by label or by AOT name.

UPDATE - the same field was also added into the "Personalize-Add columns" dialog(thanks @wojzeh for the idea)

Summary

You can download this tool using the following link https://github.com/TrudAX/XppTools#-display-system-field-name-in-the-query-filter

If you have some ideas how to improve it, feel free to create a GitHub pull request or leave a comment.

Working with Financial dimensions in D365FO

Wed, 06 Mar 2019 22:12:03 GMT

Working with the financial dimensions framework has become much more difficult starting with the AX2012. This post describes the most common problems and how to resolve them.

The current problems

The current challenges that I saw on the projects while working with the financial dimensions framework were the following:

Misunderstanding of the concept

There is no starting point if you want to learn Financial dimensions framework. The best I found is the original Implementing the Account and Financial Dimensions Framework for Microsoft Dynamics AX 2012 Applications whitepaper that explains the concept, but its content is quite outdated for D365FO. GitHub request to update this document was closed, standard D365FO Financials development home page contains some really specific information and no basic overview.

Reference to the dimension

Sometimes you need to reference just one dimension value(for example get or set "Cost center" value). You don't see many examples of this in the standard application, as in most cases dimension merging is using. In AX2009 it was quite easy - you had an enum and could use it to get or set the value. In the current version, you need somehow to reference a record in the DimensionAttribute table and there is no "recommended" way of doing this. So, I have seen different implementations:

Add DimensionAttribute to parameters(as String or as RecId)
Create a new table with enum and reference to DimensionAttribute
Search DimensionAttribute by backing table ID
Search DimensionAttribute by name defined as a Macro
... sometimes combinations of these options to access the same dimensions

All this can create a real mess, some of these options require a setup, some aren't compatible with cross-references.

Problem: Methods for working with Dimensions

There are a lot of classes to manipulate dimensions, so it is often difficult to find the right one. Moreover, classes related to dimension were renamed in D365FO(comparing to AX2012). As a result, developers sometimes create a duplicate of the existing methods.

For example, if you google "assign a value to default dimension" the first link will point to the 2-page method

Probably it can work but can cause difficulties with the support, performance and upgrades.

Proposed solution - DEVDimensionHelper class

To resolve the described issues I have created DEVDimensionHelper class(you can rename it with your project prefix or use the default DEV). The idea is to collect all the necessary stuff in one place.

In order to reference a dimension, you create a static method in this class with a default value

    static protected Name BusinessUnit   = 'BusinessUnit';
    static public Name BusinessUnit()  
    { 
        return BusinessUnit;  
    }

In this case, you don't need to perform any setup and can use "Find All References".

Currently this class contains the following references:

Create a new dimension
Query filter by dimension
Get Dimension value as string from the DefaultDimension
Get Dimension value as string from the LedgerDimension
Get a DefaultAccount from the LedgerDimension
Get a DefaultDimension from the LedgerDimension
Create a DefaultAccount from the MainAccount(String)
Create a DefaultAccount from the MainAccount(RecId)
Create a LedgerDimension from the DefaultAccount and DefaultDimension
Create a DefaultDimension from the Dim value(String)
Merge DefaultDimension with the DefaultDimension
SQL statement to get dimension(used for debug purposes)

As you see, there is no new code, all these it's just references(or small wrappers) to the standard classes. These methods will cover typical project requirements, if you need more, you can add them to this class.

The idea is to use this class as a starting point for any dimension related question on the project(even before Googling something). You get some dimension related question, you check this class, if there is no answer in it, research how to do this task and update the class with the solution(either in form of new methods or just references to existing objects).

Update 10/02/2020:

To avoid always hardcode dimension names, I have made the following adjustment in the solution:

Created a static constructor in DEVDimensionHelper class that allows overriding the default dimension names(you need to uncomment this code or add your code)

Created a simple table and a form(DEVDimNamesReference) where you can setup dimension name references(this can be used for example in situations where you have CostCenter and CostCente dimensions)

This form usage is optional, and even if it is used it requires zero setup - if the value in this form is not specified the default value from the class will be used.

Summary

You can download this class using the following link https://github.com/TrudAX/XppTools/blob/master/DEVCommon/DEVCommon/AxClass/DEVDimensionHelper.xml

If you know some other methods that can be added into this class feel free to create a GitHub pull request or leave a comment.

30 additional methods to the Global class

Wed, 27 Feb 2019 22:12:03 GMT

Global class is a standard class that contains a lot of small static functions. This post describes additional functions that extend it. To avoid using prefixes, all functions were added into the separate static class DEV. It exists in the DEVCommon model. As common practice, you can copy this class into your application as it is, or rename it according to your model name(e.g. ABC)

DEV class methods

buf2Buf

Copies one cursor to another using field names(standard buf2Buf uses IDs). So you can copy values between different tables

static void buf2Buf(Common _dest, Common _source, container _fieldsToExclude = conNull())
static void buf2BufMerge(Common _dest, Common _source)

canBeConverted, convertQty

The same functions as UnitofMeasureConverter::canBeConverted and ::convert but with the string parameters instead of RecIds

public static client server boolean canBeConverted(
        UnitOfMeasureSymbol     _fromUnitOfMeasure,
        UnitOfMeasureSymbol     _toUnitOfMeasure,
        ItemId                  _ItemId)

qty

Rounds the value according to unit settings

static Qty qty(Qty  _qty, UnitOfMeasureSymbol  _unit)

cObject

Casts parameter to Object

static Object cObject(Object  _obj)
{  return _obj;   }

countTotalQuick

Calculates the number of records in the query using Count(RecId). Standard SysQuery::countTotal switches to the loop mode when the query contains more than 1 DataSource.

static Integer countTotalQuick(QueryRun _queryRun)

copyRanges

Copies DataSource ranges

static void copyRanges(QueryBuildDataSource _qbdsTarget, QueryBuildDataSource _qbdsSource)

datasourceRangesAsText

Gets a string, that contains all ranges values. Used in reports to display current filter values in the report header.

static Notes datasourceRangesAsText(QueryBuildDataSource _sourceDS)

date2DateTime, dateTime2Date

Converts date to DateTime using UserPreferredTimeZone

static utcDateTime date2DateTime(TransDate _date, boolean _isEndOfDay = false)
static date dateTime2Date(utcDateTime _dateTime)

systemdateget

Returns the current date. It is still unclear for me, why the original method was marked as obsolete

static TransDate systemdateget()
{ return DateTimeUtil::getSystemDate(DateTimeUtil::getUserPreferredTimeZone()); }

dateStartWk, dateStartYr

Returns the start of the week, the start of the year

static TransDate  dateStartWk(TransDate _transDate)
static TransDate  dateStartYr(TransDate _transDate)

dsAllowEditExceptFields

Enable or disable all form DS fields, except specified

static void dsAllowEditExceptFields(FormDataSource _formDataSource, boolean _allowEdit, container _fieldListExclude=connull())

dsGetDisabledFields

Get a list of the disabled fields. Used with the previous function when Datasource contains some code, that disables some fields, before dsAllowEditExceptFields call.

static container dsGetDisabledFields(FormDataSource _formDataSource)

dsRefresh, dsResearch, dsExecuteQuery

Performs datasource refresh/research/executeQuery

static void dsRefresh(Common  _record)
static void dsResearch(Common _record, NoYes _savePosition = NoYes::Yes)
static void dsExecuteQuery(Common _record, NoYes _savePosition = NoYes::Yes)

getFormRunFromFormArgs

Gets the calling form from the current form args

static Object getFormRunFromFormArgs(FormRun  _element, IdentifierName _formName)

queryHasRecord

Checks, that the query has a record. The common usage scenario is when you need to create a custom lookup for the field. In this case, you create a query for this lookup, create the lookup based on this query and add a check into the validateWrite() method to check that the entered value(user can enter the value using the manual entry) exists in this query

static boolean queryHasRecord(Query  _q)

isUserInRole

Is the current user included into the role

static boolean isUserInRole(Description  _roleName)

purchTableConfirm

Confirms a purchase order

static void  purchTableConfirm(PurchTable  _purchTable)

runButtonWithRecord

Executes menu item button with the specified record

static void runButtonWithRecord(FormFunctionButtonControl _button, Common _record,
                                               Object _obj = null)

validateCondition

Validates the specified condition

static boolean validateCondition(anytype _condition, str _message, boolean _isThrowError = false)

validateCursorIsNotEmpty

Performs if (! A.RecId) {throw error("..")}

static void validateCursorIsNotEmpty(Common  _common, str _someText = '')

validateCursorField

Checks that the field is not empty and generates a message

static boolean validateCursorField(Common _table, fieldId _fieldId, boolean _isThrow = true)

validateWriteRecordCheck

Calls a validateWrite() method and generates an error if it fails

static void validateWriteRecordCheck(Common _record)

mergeInventDim

Merges 2 inventDim values

public static InventDim mergeInventDim(
        InventDimGroupSetup         _dimGroupSetup,
        InventDim                   _fromInventDim,
        InventDim                   _addedDim)

w

Displays the current value - info(StrFmt('%1',_i));, using mostly for debug purposes.

static void w(anytype _i, str _addition = '')

Summary

You can download this class using the following link https://github.com/TrudAX/XppTools/blob/master/DEVCommon/DEVCommon/AxClass/DEV.xml

If you know some other helpful functions feel free to create a GitHub pull request or leave a comment.

Fields list form(if you like Table browser, you will love it)

Wed, 13 Feb 2019 22:12:03 GMT

Fields list form is an extended version of the standard Show all fields form with the additional features such as displaying all fields with the extended information, comparing and editing.

Currently, you have the following options to directly view or edit D365FO data:

SQL via SQL Management Studio
Table browser form
Record info - Show all fields form

Typical problems when using these tools are:

Table fields that have Visible = false are not shown in D365 forms
If the table has many columns, it is hard to find the required column
D365 doesn't allow to change values for the columns that have AllowEdit=false

Fields list form provides a convenient way to view current record data with the following options:

Data is shown in the list view with Name and Label for each field(even with Visible = false) that allows you to quickly find the field value
Additional field information(like EDT name, Enum value, Enum name..) is displayed
You can view data for the current record, even if it is not saved into the database
You can compare records
You can update or delete the current record with or without validation(it is not what you should use on test/prod data, but sometimes it can be useful during the development or debugging process)

Fields list form usage

Open All sales orders form and go to the Options-Record info and then press Fields list button

Fields list form will show the current data for the selected sales order

Another way to run this form is to Right click on the form field, press Form information and then Field list button. This action doesn't save the current form cursor, so you can view the uncommitted data.

Edit feature

You can edit a field value or delete the current record. To change a field value press the Change value button.

If you check Ignore update parameter, an update will be performed using doUpdate() method. Delete provides the same option. Do not use editing feature on test/prod data, you can easily corrupt the data.

Compare feature

You can compare different records. For example, we can compare 2 sales order lines.

Select the first line and press Options - Record info - Fields list button. Then, expand the Compare group and Save the first line

Close this form, return to Sales order and select the second line. Run Fields list form again. Then you can compare previously saved Line1 and the Current record(sort by the Different column to see all differences)

Summary

Fields list tool in some cases can simplify and increase the development speed. You can download it using the following link - https://github.com/TrudAX/XppTools#installation

Feel free to post any comments(better as a GitHub issue) or ideas, what else can be improved.

Blocking in D365FO(and why you shouldn't always follow MS recommendations)

Wed, 30 Jan 2019 20:12:03 GMT

Understanding blocking is a key component of resolving performance issues. You can have fast CPU, a lot of memory, but if you face SQL blocking problem, all these will not be fully utilized. Moreover, some tips from the Performance webcasts can actually lead to blocking. In this post, I am going to describe the most common reasons for blocking and how to avoid them.

History

SQL Server engine doesn't support "true" row-level blocking, and this was a big problem in the days of Axapta 3.0. But starting from AX 4.0 and SQL Server 2005 the situation has changed. SQL Server 2005 introduced Read Committed Snapshot Isolation feature, and in AX 4.0 "Occ enabled" table property was added. Both these components significantly reduced the chance of blocking.

For more information check this article

https://cloudblogs.microsoft.com/dynamics365/no-audience/2009/07/08/about-locking-and-blocking-in-dynamics-ax-and-how-to-prevent-it/

Test example

Let's test the common blocking scenario. We will use the following table DEVBlockingTestTable , that has 4 fields, Field1 is a unique Alternate Key, Field2 is included into a non-unique index.

We create a form DEVBlockingTest with a set of actions. Initial data will be the following

We need to run 2 sessions(for OneBox VMs using the following link https://usnconeboxax1aos.cloud.onebox.dynamics.com/?f=DEVBlockingTest ) to test whether we have blocking or not. The first session will update the rows marked in red, second session - marked in green.

Test 1 - select forupdate

The first test is using select forupdate statement

For the first session enter Row number 1 = "1" and Row number 2 = "0" and press "select forupdate" button. For the second session enter "1" and "1" and press the same button

ttsbegin;
while select forupdate blockingTableLocal
    where blockingTableLocal.Field2 == updateRow1 &&
          blockingTableLocal.Field3 == updateRow2
{
    blockingTableLocal.Field4 += "u1";
    blockingTableLocal.update();
}
Box::okCancel("Select forupdate has been done", DialogButton::Ok);
ttsabort;

Results: both sessions will show an OK message. Actually, it doesn't matter, what fields or conditions we are using in the forupdate statement, sessions will not block each other in any case. Even in the worst case scenario, when 2 sessions will update the same row, blocking doesn't happen, instead, the UpdateConflict exception will be generated(this is the drawback of "Occ enabled" property, it doesn't fit for tables where concurrent updates are normal, like InventSum)

Common mistakes

Blocking is sometimes not taken into account when discussing performance best practices. One of the recent examples can be found in the "Performance Patterns and Anti-patterns" webcast, that can be found at this link: https://infopedia.eventbuilder.com/index?landingpageid=92tzhl

In this webcast, you can find the following advice(on the 17th minute)

What happens if we use this advice and switch to set-based command?

Test 2 - update_recordset

let's test this. The same example, 2 sessions, updating a different set of records, but using the update_recordset statement:

ttsbegin;

update_recordset blockingTableLocal
    setting Field4 = "u1"
    where blockingTableLocal.Field2 == updateRow1
     &&   blockingTableLocal.Field3 == updateRow2;

Box::okCancel(strFmt("update_recordset has been done. Rows: %1", blockingTableLocal.RowCount()), DialogButton::Ok);

ttsabort;

When we run the second session, we will see "Please wait.." message. So we just switched our system to "single user mode" using the best practice advice.

Ways to resolve this blocking

The best advice based on practical experience - try to avoid set-based operation where possible(this is exactly the opposite of what is recommended in this Microsoft webcast). On the next slide, they show the time of these operations. Update 1000 records one by one takes 300ms versus 6ms in the update_recordset case.

But in any case it is fast. Time to post a typical AX document with 1000 lines can be from 1 to 10 minutes. It will not be noticeable for the user either you add 6ms or 300ms to this time. But it will be very noticeable if the sessions block each other.

Another option is to create an index, that includes all fields used in the WHERE clause. You can test it using the following command.

CREATE NONCLUSTERED INDEX [I_TEST1] ON [dbo].[DEVBLOCKINGTESTTABLE]
(
    [PARTITION] ASC, [DATAAREAID] ASC, [FIELD2] ASC, [FIELD3] ASC
)ON [PRIMARY]

Field3 may be added as a normal or "include" index field. This will resolve our blocking issue in test 2. But when someone tries to update the same table using a different set of fields(for example Field2 and Field4) we'll get blocking again.

Does adding an index always remove blocking? No. Sometimes you need to remove existing index in order solve it. One of the most common cases - blocking of InventSumDelta table in AX2012 - https://axology.wordpress.com/2013/12/19/locking-on-the-inventsumdelta-table-additional-tweaks/

Locking on the InventSumDelta table – additional tweaks - We tried changing the column sequence on the two tables to have Partition and DataAreaId in the beginning, but nothing happened until we disabled the ItemId index too. That had an instant effect and practically removed the deadlock issues immediately.

Summary

Following some performance tips(even if they are given by Microsoft experts) without knowing the drawbacks can sometimes slow down your system.

Try to avoid any update_recordset and delete_from usage by default(especially in document posting operations - e.g. Sales-Purch orders, Journals..) if you don't get any measurable performance boost(less than 1%). For example, if you want to update/delete less than 1000 rows in a transaction that can last more than a minute, better to use select forupdate.

Use set based operations only when you are 100% sure that it will not cause blocking and you really need to reduce the operation time
If you still want to use update_recordset, check that all fields, used in WHERE condition, are covered by the existing indexes

I put objects used in this article to GitHub (form DEVBlockingTest, table DEVBlockingTestTable - https://github.com/TrudAX/XppTools/tree/master/DEVTutorial). Feel free to experiment with it.

Update 16/08/2021

Check this video from Brent Ozar Blocking and Locking: How to Find and Fight Concurrency Problems that explains the fundamentals of locking in detail

RunBase template class builder

Tue, 08 Jan 2019 22:12:03 GMT

This topic describes "RunBase class builder" Add-in, its parameters and usage. The purpose of this Add-in is to generate code template for a class that extends RunBaseBatch based on typical usage scenarios. Use the following link to download it.

Main screen

Add-in can be run from the Dynamics 365 - Addins - RunBase class builder

You can specify values for each parameter, preview the changes(using the Refresh button) and if you are fine with it, automatically generate the class and the related menu item.

As a result, the following class will be generated

Let's describe the Add-in parameters

-Class name

The name of the resulting class.

-Description

The value that will be returned from the static description() method.

-Parameters..

You can specify several parameters that will be displayed in the user dialog, one line per parameter. To describe the parameter the following information should be provided(values are separated by a | delimiter):

EDT - Extended data type name for the dialog field. For mandatory fields add * at the end
Variable name - class variable for this EDT. Can be empty, in this case, it will be generated from the EDT name
Label - you can override EDT label for the field on the dialog form. If the Label is empty, EDT label will be used
Help text- you can override EDT help text for the field on the dialog form. If it is empty, EDT help text will be used

The following code is generated for each parameter:

//ClassDeclaration
CustAccount  custAccount;
DialogField  dlgCustAccount;
#localmacro.CurrentList
    custAccount
#endmacro
//dialog
dlgCustAccount     = dialog.addFieldValue(extendedtypestr(CustAccount), custAccount);
//getFromDialog
custAccount      = dlgCustAccount.value();
//parm method
public CustAccount parmCustAccount(CustAccount _custAccount = custAccount)...

-Query table

If you want to display a query in the dialog, you need to specify the main table for this query. In this case, the following code will be added:

//ClassDeclaration
QueryRun       queryRun;
//initQuery() sample code for query init
Query                   query = new Query();
QueryBuildDataSource    qbds = query.addDataSource(tablenum(CustTable));
QueryBuildRange         qBR;
;
qBR = SysQuery::findOrCreateRange(qbds, fieldnum(CustTable, RecId));
qBR.status(RangeStatus::HIDDEN);
queryRun = new QueryRun(query);
//pack() unpack() support for this queryRun
//run() - sample code
qBDS = queryRun.query().dataSourceTable(tableNum(CustTable));
        SysQuery::findOrCreateRange(qBDS, fieldnum(CustTable, RecId)).value(queryValue(""));

this.progressInit(RunBase::getDescription(classIdGet(this)),
                  SysQuery::countTotal(queryRun),
                  #AviSearch);
while (queryRun.next())
{
    custTable = queryRun.get(tablenum(CustTable));

    processedCounter++;
    progress.incCount();
}

-External table name

Table name of the caller record should be specified. This covers a scenario when class is calling from a Menu item button on a form. Often, in this case, we need to save the caller record, to check for the caller record or to update the existing query based on the caller record.

The following code will be generated (External table name = CustTable)

//ClassDeclaration
CustTable    callerCustTable;
//main
if (_args.dataset() != tablenum(CustTable))
{
    throw error(Error::missingRecord(funcname()));
}
runObject.parmCallerCustTable(_args.record());
if (_args && _args.record().TableId == tablenum(CustTable))
{
    qbds = runObject.queryRun().query().dataSourceTable(tablenum(CustTable));
    qbds.clearRanges();
    qbds.addRange(fieldnum(CustTable, RecId)).value(queryValue(runObject.parmCallerCustTable().RecId));
    //runObject.parmIsDisableUnpackQuery(true);
}

-Add file upload

You need to specify 'y' symbol to generate code that allows a user to select a file from the local PC. It will add a fileUploadResult variable, a necessary code to handle this file and a sample code that displays uploaded text file content.

//run()
memoryStreamFile = fileUploadResult.openResult();
AsciiStreamIo asciiIo = AsciiStreamIo::constructForRead(memoryStreamFile);
asciiIo.inRecordDelimiter('\n');
while (asciiIo.status() == IO_Status::Ok)
{
    container c = asciiIo.read();
    if (conLen(c) > 0)
    {
        info(strFmt("File data:%1", conPeek(c, 1)));
    }
}

Comparing to AX2012(where you can simply define a variable with the FileNameOpen type) adding file upload button requires more efforts in D365FO, as Dialog doesn't include new file control. I even created an idea to improve this https://experience.dynamics.com/ideas/idea/?ideaid=ac0834aa-216c-e711-80c0-00155d7cd0b4, but its status is still New.

Specify 'y' symbol to create an Action menu item, linked to the created class.

Summary

This utility covers some common patterns of RunBase usage, so you can focus on business logic developing rather than typing standard code. If you find that something is missing or can be improved don't hesitate to leave a comment.

How to setup Financial reporting on D365FO local One box VM

Sat, 22 Dec 2018 22:12:03 GMT

Using Financial reporting (aka Management Reporter in Ax2012) on local DEV D365FO one box requires some addition settings, that are not yet documented. This post describes the problem and possible solution.

Problem description

Let's start with the standard 8.1PU20 VM One box.

Download and install it, rename and execute AdminUserProvisioning tool.

Then open General ledger - Inquiries and reports - Financial reports and press New in this form

Go to the Tools - Reset Data Mart and press OK.

You will get the error "There was a problem communicating with the server"

and weird text in the Event log, like this

exceptionMessage {"Class":16,"Number":8169,"State":2,"Message":"Conversion failed when converting from a character string to uniqueidentifier.","ErrorCode":-

There is a standard topic about Data Mart reset on the docs site, but it works only if you have a proper connection properties

https://docs.microsoft.com/en-us/dynamics365/unified-operations/dev-itpro/analytics/reset-financial-reporting-datamart-after-restore?toc=/fin-and-ops/toc.json

If your search the same topic for AX2012, you will find several posts, that describe full reset(like this one - Management Reporter: Rebuilding your Data Mart https://community.dynamics.com/365/financeandoperations/b/dynamicsaxbi/archive/2016/02/01/management-reporter-rebuilding-your-data-mart )

But the problem is that in D365FO we don't have MR Configuration Console, instead, PowerShell should be used.

Solution

Let's recreate a Data Mart using the standard PowerShell scripts

Open PowerShell ISE as administrator

As a first step, let's back up our existing ManagementReporter database

$FileNameLocal = "C:\backup\ManagementReporter.bak"
$FileDirName = Split-Path $FileNameLocal
New-Item -ItemType Directory -Path $FileDirName -Force
Backup-SqlDatabase -ServerInstance localhost -Database "ManagementReporter" -BackupFile $FileNameLocal -CompressionOption On

Then import the standard MRDeploy library and check it's default settings

cd C:\FinancialReporting\Server\MRDeploy
Import-Module –Name C:\FinancialReporting\Server\MRDeploy\MRDeploy.psm1 -Verbose
Get-MRDefaultValues

The current problem is that many pre-configured parameters contain irrelevant to local One box values(like DB name is AXDBRAIN instead of AxDB, some users credentials are empty)

I could not find any documentation that describes the correct configuration for One box, so just specified one value for all user accounts.

Set-MRDefaultValues -SettingName AXDatabaseName -SettingValue AxDB
Set-MRDefaultValues -SettingName AosUserName -SettingValue AOSUser
Set-MRDefaultValues -SettingName AosUserName -SettingValue AOSWebSite@123
Set-MRDefaultValues -SettingName AosWebsiteName -SettingValue AOSService
Set-MRDefaultValues -SettingName MRSqlUserName -SettingValue AOSUser
Set-MRDefaultValues -SettingName MRSqlUserPassword -SettingValue AOSWebSite@123
Set-MRDefaultValues -SettingName MRSqlRuntimeUserName -SettingValue AOSUser
Set-MRDefaultValues -SettingName MRSqlRuntimeUserPassword -SettingValue AOSWebSite@123
Set-MRDefaultValues -SettingName DDMSqlUserName -SettingValue AOSUser
Set-MRDefaultValues -SettingName DDMSqlUserPassword -SettingValue AOSWebSite@123
Set-MRDefaultValues -SettingName DDMSqlRuntimeUserName -SettingValue AOSUser
Set-MRDefaultValues -SettingName DDMSqlRuntimeUserPassword -SettingValue AOSWebSite@123
Set-MRDefaultValues -SettingName AXSqlUserName -SettingValue AOSUser
Set-MRDefaultValues -SettingName AXSqlUserPassword -SettingValue AOSWebSite@123
Set-MRDefaultValues -SettingName AXSqlRuntimeUserName -SettingValue AOSUser
Set-MRDefaultValues -SettingName AXSqlRuntimeUserPassword -SettingValue AOSWebSite@123

The main script can then be run. It will delete the existing ManagementReporter database and performs a new DB initialization.

New-MRSetup  -IntegrateDDM

After that, you can run the Reset command without errors. If you check Integration status you will see "Integration complete"

Summary

I posted the whole script to GitHub - https://github.com/TrudAX/TRUDScripts/tree/master/FinancialReporting It has been tested for the 8.1 version. If you find any issues or have ideas how to improve it, Pull requests are always welcome. Hope Microsoft will fix this in the new version, as right now it looks like a missing documentation step.

Do we really need index hints? (forceLiterals vs forcePlaceholders)

Mon, 10 Dec 2018 22:12:03 GMT

The idea to write this post came after a discussion of the new PU23 feature “Enabling index hints in X++ again”

https://docs.microsoft.com/en-us/dynamics365/unified-operations/fin-and-ops/get-started/whats-new-platform-update-23#enabling-index-hints-in-x-again

And the main question during the discussion was – Do we really need “index hint” for the modern SQL Server?

First, let's discuss the basic methods of SQL query generation. In D365FO we have 2 ways of executing a SQL statement

ForcePlaceholders

This is the default query execution mode. In this case, every SQL Statement that you write in X++, is processed in 2 steps – first sp_prepare command executed with placeholders instead of actual values and as a result of this sp_prepare command, we get a cached plan. Then sp_cursorexecute is called, that executes the query using this plan and the actual values. For one SQL statement with the different values, the plan is calculated only once.

How does SQL server determine what plan to generate in sp_prepare if the statement is new (for some values one plan may be more efficient than for others)? For new unknown plans it uses a “parameters sniffing” feature. If it gets sp_prepare command for a new statement, it doesn’t try to calculate it immediately, instead, it waits for the first call of sp_cursorexecute and uses the first passed values to calculate the plan.

ForceLiterals

In this case, your current SQL statement (with the actual values) is sent to SQL Server and executed as it is. The drawback of this is that for every set of values you will need to parse this statement and compile a new SQL plan.

Parameters sniffing

Parameters sniffing quite often becomes a reason for performance problems. For example, you have a warehouse where most of the items have one batch ID, but there are some Items and Batches with general names (like “No batch”, “Empty”). In this situation, when you query item by batch your actual plan depends on the first query values. If your query contains “general” batch values, SQL Server creates a plan that starts execution with the ItemId, as a “general batch value” is not selective in this case. But for most of your items, it will be wrong and gives huge overhead, as in most cases Batch is a unique value.

How to resolve a Parameters sniffing issue:

There is no universal way to resolve Parameters sniffing issue(refer to the excellent BrentOzar post that describes this https://www.brentozar.com/archive/2013/06/the-elephant-and-the-mouse-or-parameter-sniffing-in-sql-server/ ), but there are several ways to deal with it in D365FO

Create new indexes – you can create new indexes, to help SQL server always choose the best plan. This often works only if you have conditions to one table only
Use the sp_create_plan_guide command to force the actual plan – using this option creates a big admin overhead as you need to maintain these created plans. If you add a new field to the AX table you will need to change all the plan guides in which this table is used
forceLiterals hint –you send actual values to the SQL Server and it chooses the best plan for these values. The overhead is that the plan will need to be compiled every time
Use index hint (new feature in PU23) – it is the same as sp_create_plan_guide but with no admin overhead

Overhead of forceLiterals usage

Let’s check, what is the overhead of forceLiterals usage(I have tested this on medium size AX2012 DB and on D365 8.1 demo, time proportions were the same in both cases)

For the test let’s use the following job. It calculates actual item quantity for the warehouse.

class dev_testForceLiterals
{
    public static void main(Args _args)
    {
        InventTable         inventTable;
        InventDim           inventDim;
        InventSum           inventSum;
        InventLocation      inventLocation;
        int                 curCount, maxCount = 5000;
        System.Diagnostics.Stopwatch stopwatch = new System.Diagnostics.Stopwatch();
        real                         timeMs;
        ;
        stopwatch.Start();
        while select inventTable
        {
            while select inventLocation
            {
                //select forcePlaceholders  //case 1
                select forceLiterals        //case 2
                    sum(AvailPhysical) from inventSum
                group by ItemId
                where inventSum.ItemId == inventTable.ItemId &&
                      inventSum.Closed == false
            exists join inventDim
                where inventDim.inventDimId      == inventSum.InventDimId &&
                      inventDim.InventLocationId == inventLocation.InventLocationId &&
                      inventDim.InventSiteId     ==   inventLocation.InventSiteId;

                curCount++;
                if (curCount > maxCount) break;
            }
            if (curCount > maxCount) break;
        }
        stopwatch.Stop();
        timeMs = stopwatch.get_ElapsedMilliseconds();
        info(strFmt("Time for %1 lines: %2 sec", curCount, timeMs / 1000));
    }
}

Before each run we will clear SQL plan cache using the following command (note that this command only clears plan cache, not the data cache, all used tables are still located in the memory)

DBCC FREEPROCCACHE
GO

To calculate memory consumption after each run we will use the following statement

SELECT objtype AS [CacheType]
        , count_big(*) AS [Total Plans]
        , sum(cast(size_in_bytes as decimal(18,2)))/1024/1024 AS [Total MBs]
        , avg(usecounts) AS [Avg Use Count]
        , sum(cast((CASE WHEN usecounts = 1 THEN size_in_bytes ELSE 0 END) as decimal(18,2)))/1024/1024 AS [Total MBs - USE Count 1]
        , sum(CASE WHEN usecounts = 1 THEN 1 ELSE 0 END) AS [Total Plans - USE Count 1]
FROM sys.dm_exec_cached_plans
GROUP BY objtype
ORDER BY [Total MBs - USE Count 1] DESC

forcePlaceholders results

For the first and for the second executions, time was the same

First run: 8,51 sec

Second run: 8,51 sec

Memory consumption:

forceLiterals results

First run: 66,57 sec

Second run: 9,60 sec

Memory consumption:

During the execution there was almost 100% utilization for 1 CPU core

If we present these values on a graph:

Memory consumption is about 100KB per saved plan. So in forceLiterals case, it requires a 500MB memory per 5k SQL statements. This is a large amount of memory just for the plan cache.

Summary

Plan compilation is still a complex task, that consumes a lot of CPU. This 1-minute difference between the first run of forceLiterals and forcePlaceholders is utilizing one CPU core for 100%. For the forceLiterals case, time to build a plan is 6 times longer than the time to fetch the actual data. Also, notice a difference between the second run of forceLiterals and forcePlaceholders. It is about a second – this time is a time server spends doing lexical parsing of SQL statement.

For single SQL statements (or statement that executes once per document header) forceLiterals can provide some benefit, but if we have a large number of users who are working with different items and different dimensions forceLiterals usage for the frequent SQL requests can affect overall SQL server performance (both for memory consumption and CPU), so we need to avoid forceLiterals usage on a "per line" level. And the new “index hint” feature can help us in this.

Gatsby RSS Feed

X++ Show Cross-Reference With Read/Write field reference

D365FO License usage log utility

Understanding the current licensing model

Special roles

ISV solutions related to licensing

License usage log tool

Enabling element usage logs

Calculating data modifications

Service functions

Running the license usage report

Header section

Lines section

Analyzing the license report data

1. Match

2. Match – Write Not Confirmed

3. Higher Than Required

4. Higher Than Required (No Activity)

Quick overview with AI

Summary

Implement Service-based integration in D365FO

Key Design Principles

Implementation details: X++ code

Implementation details: User configuration

Logging

Notes on consuming services

Summary

Sync data from Dynamics 365 Finance & Operations Azure SQL Database (Tier2) to local SQL Server (AxDB)

Usage

Custom Copy Strategies

Examples

Copy tables where a row contains large data

Copy Inventory dimensions if Licence plate is used(InventDim)

Copy inventory onhand (InventSum)

Copy inventory reservations

Copy Performance Optimization

SysRowVersion Optimization (Incremental Sync)

Smart Strategies

Execution Efficiency

Future Usage Scenarios

How This Tool Was Built

Summary

D365FO Integration: Import Purchase Orders from PDF using Gemini AI

Integration Scenario Overview

Current solutions on the market

PDF to Excel converters

Invoice capture solution

Use AI models

External integration AI solution setup

AI Providers

Connection type

AI Prompt Definition

Inbound message type

Operation parameters

Main modification description

Invoice validation

Purchase order creation

Resources for This Blog Post

Summary

D365FO Integration: Import Sales Orders from an External Web Application

Integration Scenario Overview

Planning and Scoping Integration Project

Define the Web API Endpoint and Parameters

Confirm the Scope of Returned Documents

Establish Rules for Handling Updated Documents

Discuss the TEST Endpoint

Request a Guide for Creating Documents via the User Interface

Resolving Mapping Errors

Create a Data Mapping document

High-level Design Description

Setting Up the Integration

Configuring Connection Types

Defining Inbound Message Types

Mapping Setup

Running the Integration

Generating Test Data

Running an Incremental Load

Monitoring and Error Handling

Connection Errors

Parsing Errors