AX user sessions hanging following changes to security
Introduction
In this article I describe behaviour that can occur in AX 2012 and AX 2012 R2 environments when changes are introduced to a production system that affect security. The behaviour has been termed “security-cache thrashing”. As well as giving a detailed explanation about how this situation can arise, I also discuss some best practices that will ensure that security cache thrashing can be avoided in production environments. Finally, I have summarised enhancements we have introduced since the original release of AX 2012 relating to how cached security metadata is created and maintained.
Symptoms
Security cache thrashing can cause a number of different performance issues. A common symptom of this problem is when multiple user sessions becoming unresponsive. This can then lead to high memory usage by the AOSs and increasing numbers of online user sessions, as affected users abandon their hanging session and try to initiate further sessions on the server. New user logons may eventually be blocked, making the system completely inaccessible.
When running an AX trace or monitoring database activity with SQL Server tools, you may observe that the stored procedure “XU_SecurityLoadRoles” is running for a long duration. You may also see the message “Duplicate request to load role permissions detected” reported in the application event logs of the AOS server.
Background
The introduction of role-based security in AX2012
In AX 2012, we introduced a new security model based on roles. A role is primarily a container for privileges, with each privilege defining the level of access a user has to different parts of the system. These privileges may be grouped together and assigned to the role as a duty, or they may be assigned individually.
As well as being a container for a set of privileges, a role can also:
- contain other roles within it, thereby inheriting all the permissions defined for that “sub-role”
- define its own set of permissions directly on individual forms, tables and server methods.
The Security Cache
When a user logs into AX for the first time, the system must extract all of the different permissions that they are granted via the security role(s) they belong to. It only needs to do this once, as it caches the information when it is extracted. However, to ensure that users are always granted the correct level of access to different parts of the system, the caches must also be updated whenever changes are made that affect security.
A cache is first created in the TempDB database. Storing a centralized cache allows for the data to be used by other AOSs in the environment, as well as ensuring that the information is retained if AOSs are restarted. Once the cache is built in the database, individual AOSs will then load the information into memory to avoid unnecessary calls to the database.
Cause
During the time the security cache is being rebuilt, users may find that their AX client hangs (i.e. becomes unresponsive). This happens if the system needs to determine whether the user has access rights to a particular object – it can’t do this until the security permissions for the user’s roles have been reloaded into the database security cache and subsequently into the AOS’s memory. Likewise, if new users attempt to log on, their clients may hang during start-up if the cache is being repopulated with data for any roles those new users are members of.
Actions that can affect the security cache
- Changes to securable objects in the AOT
The following objects can all have permissions defined on them. If a developer makes a change to any one of these objects (for example, they add a new control to a form), the cache is invalidated and must be updated:
- Forms
- Menu items
- SSRS Reports and X++ Reports
- Web Components
- Service Operations
It is not recommended to implement changes into a live environment whilst users are accessing the system. If best practices regarding the deployment of customizations are followed, this issue will be avoided.
For further information about these best practices, please refer to the “Deploying Customizations across AX 2012 Environments” whitepaper.
2. Modifying roles/duties/privileges via the System Administration module
Security administrators have the ability to modify roles and other security objects via forms in the System Administration module. For example, in the Security Privileges form (System Administration > Setup > Security > Privileges), an administrator can modify the access level granted by a specific privilege.
We recommend against applying security changes during normal business hours to avoid invalidating the security cache when users are accessing the system.
Security Enhancements
We have introduced some changes to the behaviour of security cache flushing, in order to help reduce the impact that invalidating the cache can have on the performance of a live system.
- Cache flushing is more granular (AX 2012 R2 and AX 2012 with kernel 6.0.1108.4556)
In early versions of AX 2012, the entire database cache for all roles would be invalidated following developer changes to securable objects. This meant that permissions for every role had to be re-loaded into the database cache and subsequently into memory on each AOS. For a standard AX 2012 environment with just the out-of-the-box roles, this could take around 5 minutes to complete. For customers with a more complex security requirements, this could take significantly longer (in some cases, hours rather than minutes).
The cache is now invalidated in a more granular way. Cached permissions data is flushed per role; AX identifies which roles grant/deny access to the modified object and then only flushes the cached data for those roles.Roles are also individually flushed for changes to Roles, Duties, Privileges and Process Cycles (this has not changed since AX 2012 RTM). However, as privileges can belong to multiple roles, changing them can result in the cache being flushed for multiple roles.
- Prime the database cache using AXUtil (R2 with kernel 6.2.1000.749/AX 2012 with kernel 6.0.1108.4556)
We have also introduced functionality into AXUtil, allowing an administrator to prime the database security cache before users begin to access the system. The AXUtil command “refreshrolecache” can be used during periods of downtime to flush the database cache and recreate it, prior to starting the AOSs. This functionality was introduced into AX 2012 R2 via hotfix KB2832056, which updates the kernel to build 6.2.1000.749. The same functionality was also back-ported to AX 2012 and is included in kernel builds 6.0.1108.4556 and above.
