Showing posts with label CIL. Show all posts
Showing posts with label CIL. Show all posts

Monday, February 25, 2019

TempDB table populated in CIL

We have multiple options to provide additional information to records in grids. Some of them, like display methods, are easy to implement. Some of them, like a temporary table, are not.

Let's consider a scenario when our lovely user wants to see reservation info directly in the transaction grid as filterable columns.






Evidently, we cannot use display methods because they cannot be filterable. Given the way this info calculated behind the curtain, there is no way to create a view to join InventTrans on InventTransItem form. So, the last feasible option is to create a temporary table and join it to the main form data source.

In order to populate this TempDB table (it is not enough to create an InMemory one) as much fast as possible, we have to run the logic in CIL.

The following shows how to work with such tables by passing their physical names.

First, create a TempDB table.





Then add it to the form data source so that it would be outer joined to InventTrans table





and place its Init() method, which links the local buffer with a populated TempDB on the SQL server side.

We also need to init its populating engine and calculate its data at least once when the form opens and maybe later via Refresh button for all or one particular item.

final class FormRun extends ObjectRun
{
    myInventSumFormEngine              myInventSumFormEngine;
}

void init()
{
    super();
    element.myInitParameters();
}

private void myInitParameters()
{
    myInventSumFormEngine               = new  myInventSumFormEngine(element);
}

//myTmpInventSum_DS.Init
public void init()
{
    myTmpInventSum                  myTmpInventSumLoc;
    myTmpInventSumLoc.doInsert();
    delete_from myTmpInventSumLoc;
    super();
    myTmpInventSum.linkPhysicalTableInstance(myTmpInventSumLoc);
}

public void run()
{
    super();
    element.myRecalcData();
}

public void myRecalcData(boolean _forCurrentItemOnly = false)
{
    myInvenTransByItemDimView  myInvenTransByItemDimView;

    if(_forCurrentItemOnly)
    {
        myInventSumFormEngine.parmItemIdCalcFor(InventTrans.ItemId);
        myInventSumFormEngine.run();
    }
    else
    {
        myInventSumFormEngine.parmItemIdCalcFor('');
        // todo: uncomment if needed to measure the execution time
        //myInventSumFormEngine.parmShowCalcInfo(true);
        startLengthyOperation();
        myInventSumFormEngine.run();
        endLengthyOperation();
    }
}

Now let's take a look at the populating class. It has parameters for showing elapsed time and run for a particular item id only.


/// <summary>
/// Populating TempDB tables in CIL
/// </summary>
/// <remarks>
/// Passing physical table names between CIL and form
/// </remarks>

public class myInventSumFormEngine
{
 // add more data sources to support multiple TempDB tables populating
 FormDataSource                      fdsTmpInventSum;
 myTmpInventSum                     myTmpInventSum;
 ItemId                              itemIdCalcFor;
 boolean                             showCalcInfo;
}

/// <summary>
/// Gets TempDB tables physical names
/// </summary>
/// <returns>
/// Container with names
/// </returns>
/// <remarks>
/// Creates buffers if not exist yet
/// </remarks>
private container getTempTablesNames()
{
 if(!myTmpInventSum.getPhysicalTableName())
 {
  select firstOnly myTmpInventSum;
 }
 return [myTmpInventSum.getPhysicalTableName()];
}

/// <summary>
/// Creates new populating engine for a given form
/// </summary>
/// <param name="_formRun">
/// Caller form
/// </param>
/// <remarks>
/// Looks for a particular data sources to link TempDB tables
/// </remarks>
public void new(FormRun _formRun = null)
{
 Object formObject = _formRun;

 if (_formRun)
 {
  fdsTmpInventSum      = _formRun.dataSource(tableId2Name(tableNum(myTmpInventSum)));
 }
}

/// <summary>
/// To calculate TempDB table for a particular item only
/// </summary>
/// <param name="_parm">
/// Item id to calculate data
/// </param>
/// <returns>
/// Item id
/// </returns>
/// <remarks>
/// If empty, then calculate for all items
/// </remarks>
public ItemId parmItemIdCalcFor(ItemId _parm = itemIdCalcFor)
{
 itemIdCalcFor = _parm;
 return itemIdCalcFor;
}

/// <summary>
/// Show consumed time to populate TempDB tables
/// </summary>
/// <param name="_parm">
/// Shows if True
/// </param>
public boolean parmShowCalcInfo(boolean _parm = showCalcInfo)
{
 showCalcInfo = _parm;
 return showCalcInfo;
}

public void run()
{
 str                     myTmpInventSumTempDBTableName;
 // add here more tempDB tables if needed
 [myTmpInventSumTempDBTableName] = this.getTempTablesNames();
 [myTmpInventSumTempDBTableName] = myInventSumFormEngine::calc([itemIdCalcFor, myTmpInventSumTempDBTableName, showCalcInfo]);
 // push calculated data back to the form data source
 if(fdsTmpInventSum)
 {
  // this assignment works only on the client tier outside of the populating method
  fdsTmpInventSum.cursor().linkPhysicalTableInstance(myTmpInventSum);
  fdsTmpInventSum.research();
 }
}

/// <summary>
/// Launcher for CIL
/// </summary>
/// <param name="_cont">
/// Container with parameters and all TempDB tables physical names to populate
/// </param>
/// <returns>
/// Container with all populated TempDB tables physical names
/// </returns>
private static server container calc(container _cont)
{
 container               cont;
 XppILExecutePermission  xppILExecutePermission;
 FromTime                startTime               = timeNow();
 ItemId                  itemIdCalcFor           = conPeek(_cont, 1);
 boolean                 showCalcInfo            = conPeek(_cont, 3);

 xppILExecutePermission = new XppILExecutePermission();
 xppILExecutePermission.assert();

 cont = runClassMethodIL(
         classStr(myInventSumFormEngine),
         staticMethodStr(myInventSumFormEngine, calcIL),
         _cont
        );
 CodeAccessPermission::revertAssert();

 if(showCalcInfo)
 {
  info(strFmt("Refreshed for %1 in %2",  itemIdCalcFor ? itemIdCalcFor : "all items", timeConsumed(startTime, timeNow())));
 }

 return cont;
}

/// <summary>
/// Implements the calculating logic and populating TempDB tables
/// </summary>
/// <param name="_con">
/// Container with parameters and all TempDB tables physical names to populate
/// </param>
/// <returns>
/// Container with all populated TempDB tables physical names
private static server container calcIL(container _con)
{
 WHSInventReserve                    whsInventReserve;
 myTmpInventSum                      myTmpInventSum;
 ItemId                              itemIdCalcFor                   = conPeek(_con, 1);
 str                                 myTmpInventSumTempDBTableName  = conPeek(_con, 2);
 myInvenTransByItemDimView           myInvenTransByItemDimView;
 // Link to an exiting table
 myTmpInventSum.useExistingTempDBTable(myTmpInventSumTempDBTableName);
 // empty existing data
 myInventSumFormEngine::cleanTmpInventSum(myTmpInventSum, itemIdCalcFor);
 /***** Implements the calculating logic here *****/
 while select whsInventReserve
  join RecId from myInvenTransByItemDimView
  where
   (!itemIdCalcFor || whsInventReserve.ItemId == itemIdCalcFor) &&
   myInvenTransByItemDimView.ItemId == whsInventReserve.ItemId &&
   myInvenTransByItemDimView.InventDimId == whsInventReserve.InventDimId
 {
  myTmpInventSum.clear();
  myTmpInventSum.Itemid                  = whsInventReserve.itemId;
  myTmpInventSum.InventDimId             = whsInventReserve.InventDimId;
  myTmpInventSum.AvailOrdered            = whsInventReserve.AvailOrdered;
  myTmpInventSum.AvailPhysical           = min(whsInventReserve.displayPhysAvailUpHierarchy(), whsInventReserve.AvailPhysical);
  // here you can add any other method calls to get more info about reservation
  myTmpInventSum.AvailReservDelta        = myTmpInventSum.AvailPhysical - myTmpInventSum.AvailOrdered;
  myTmpInventSum.insert();
 }
 // send tables physical names back to the form
 return [myTmpInventSum.getPhysicalTableName()];
}

/// <summary>
/// Empties the whole table or for given item id only
/// </summary>
/// <param name="_myTmpInventSum">
/// TempDB table buffer to empty
/// </param>
/// <param name="_itemIdCalcFor">
/// Item id
/// </param>
static private void cleanTmpInventSum(myTmpInventSum _myTmpInventSum, ItemId _itemIdCalcFor = '')
{
 delete_from _myTmpInventSum
 where
  (!_itemIdCalcFor || _myTmpInventSum.ItemId == _itemIdCalcFor);
}

As you can see the tricky point is to provide tables physical names to CIL and back in calc() and calcIL() methods.

Now you get it.




Do not forget to build CIL and check it in your Development user options.


Saturday, February 10, 2018

Why my batch job is still waiting?

You probably have compilation errors in CIL. Try to recompile full CIL.

Tuesday, February 6, 2018

Why your method is still not running in CIL

Please do not forget about the third parameter when you call your CIL wrapper.




It won't run in CIL if the current transaction level is not equal to zero, like in the following example with the standard import of Zip codes.



However, you can try to force ignoring TTS level at your own risk.


It works perfectly for my scenario.


This is how to check if your code is running in CIL.

Friday, February 2, 2018

Parallel Items/Product Validating or Deleting

If you need to validate or delete items/products or any other records in a BIG number, it is better to run such processing, first, in CIL, second in parallel threads.

This project is to demonstrate this approach.

The whole concept is similar to what I explained in one of my previous blogpost about Multi thread parallelism and a dispatching table for finding a minimum

myInventItemProcessBatch class populates a special table containing RecIds to be processed and thread number they belong to.



Based on the user selection, it creates appropriate number of batch tasks that can run independently with their progress percentage.




Feel free to elaborate this project by adding new types of processing or new table to process. Also it is probably a good idea to add a new column to the table to separate different instances myInventItemProcessBatch simultaneously running in the same environment.


myInventItemProcessBatch 

private static server int64 populateItems2Process(str 20 _what2find, int _batchThreads)
{
    myInventItemProcessTable    myInventItemProcessTable;
    InventTable                 inventTable;
    int                         firstThread = 1;
    Counter                     countr;
    // flush all previously created items from the table
    delete_from myInventItemProcessTable;
    // insert all needed items in one shot. this part can be refactored to use Query instead
    insert_recordset myInventItemProcessTable (threadNum, ItemRecId, ItemId)
    select firstThread, RecId, ItemId from InventTable
    where
        inventTable.itemId like _what2find;
    // now group them in threads by simply enumerating them from 1 to N
    countr=1;
    ttsBegin;
    while select forUpdate myInventItemProcessTable
    {
        myInventItemProcessTable.threadNum = countr;
        myInventItemProcessTable.update();

        countr++;

        if(countr > _batchThreads)
        {
            countr=1;
        }
    }
    ttsCommit;
    // return the total number of items to process
    select count(RecId) from myInventItemProcessTable;

    return myInventItemProcessTable.RecId;
}

public void run()
{
    // get all required items by their RecIds in the table and group them in threads
    int64 totalRecords = myInventItemProcessBatch::populateItems2Process(what2find, batchThreads);
    if(totalRecords)
    {
        info(strFmt("Found %1 items like '%2' to %3", totalRecords, what2find, processType));
        // create number of batch tasks to parallel processing
        this.scheduleBatchJobs();
    }
    else
    {
        warning(strFmt("There are no items like '%1'", what2find));
    }
}
myInventItemProcessTask process()

... 
select count(RecId) from inventTable
        exists join myInventItemProcessTable
        where
            myInventItemProcessTable.ItemRecId == inventTable.RecId &&
            myInventItemProcessTable.threadNum == threadNum;
    // total number of lines to be processed
    totalLines = inventTable.reciD;
    // to enjoy our bored user during a few next hours
    // this progress just updates percentage in Batch task form
    progressServer = RunbaseProgress::newServerProgress(1, newGuid(), -1, DateTimeUtil::minValue());
    progressServer.setTotal(totalLines);

    while select inventTable
        exists join myInventItemProcessTable
        where
            myInventItemProcessTable.ItemRecId == inventTable.RecId &&
            myInventItemProcessTable.threadNum == threadNum
    {
        progressServer.incCount();

        try
        {
            // RUN YUR LOGIC HERE //////////////////////
...

Wednesday, November 16, 2016

Automate Error 351

From time to time we get the lovely CIL compilation Error:351. As suggested by many, for example, by André Arnaud de Calavon, we have to recreate XppIL folder with all its guts.

The sequence is the following (quotation)

1. Stop the AOS.
2. Rename the XppIL folder (C:\Program Files\Microsoft Dynamics AX\60\Server\MicrosoftDynamicsAX\bin\XppIL) to e.g. XppIL_old.
3. Start the AOS.
4. Perform a full CIL generation.
A new XppIL folder will be created when you start the AOS.
When the CIL has completed without problems, you can delete the renamed XppIL_old folder.

Clean and simple. However, I am too impatient to wait for the end of deleting this huge folder: Windows Explorer starts time estimation and it drags on.

So, I wrote two short batch files that can be run as administrator and spare your time and nerves.

First just to rename the existing folder.


@echo off
set folder="C:\Program Files\Microsoft Dynamics AX\60\Server\CHR_AX_DEV\bin\"
echo Rename %folder%  XppIL to XppIL_old?
pause
c:
cd %folder% 
ren XppIL XppIL_old
echo %folder%XppIL to  has been renamed to XppIL_old
pause

Second to delete all the files in the 'backed up' folder and its subfolders with no questions and infolog, then to delete the folder itself. As it said here, they must work faster than just removing the folder.

@echo off
set folder="C:\Program Files\Microsoft Dynamics AX\60\Server\CHR_AX_DEV\bin\XppIL_old\"
echo Delete all the files from %folder% and remove the folder?
pause
del /f/s/q %folder% > nul
rmdir /s/q %folder%
echo Folder %folder% removed
pause

The last remarque. Be sure that your service windows account running the AOS in question had Full permission to the C:\Program Files\Microsoft Dynamics AX folder and all its subfolders.

command line

Saturday, March 12, 2016

Multi thread parallelism and a dispatching table for finding a minimum

In my free time I enjoy by solving programming puzzles from Advent of Code website. Some of them are pretty simple, though others could be tricky, however, all of them are always witty. Of course, I do it in AX so that I could use as much its power as possible.

The day 4 Ideal Stocking Stuffer became a die-hard to me. And it is not because of its "business complexity" -- you simply need to find the lowest positive number producing an MD5 hash for a given secret code, so that such a hash, in hexadecimal, starts with at least five zeroes.

Honestly, I have a vague idea about MD5 hash math -- I just took a working example and injected it into my class.

The stumbling point here was calculation time. Even for the first part of the puzzle, which is always easier than than the second one, it took so much time that I started flirting with the idea to improve performance.

Wrapping the MD5 hash calculation method so that it could be run in CIL got it faster but not enough to be happy.



The next idea was batch task execution in parallel threads, like it is brilliantly explained by Ganas1 in four chapter blog series:

Batch Bundling

However, we need to find the lowest positive number; therefore, we do not know how many tasks must be created. (Let's assume that we are limited with the maximum of Int64)


My solution is the following.

I created a table, which is to centrally dispatch creating, executing, and stopping batch tasks based on a separate, sequentially assigned positive number ranges. So, for each batch task it keeps the assigned thread number, ranges, execution status and found results, if any.



The batch task generating class creates them for a given number of logical processors, four in my environment.


Each task checks the table for a found result. If it is already found in any range, it stops.
If not, it looks for the highest range from the table and than tries to add a new record. In case of success, it runs finding the lowest number in the given range.

If such a number is found in the current thread, this value becomes a new candidate only if there are no results found in the lower ranges and no any lower ranges still running.


Now blood runs faster: even the second part of the job did not give me a pause to get another beer from the fridge.

However, it is up to your judgement to set up the right range size and number of parallel tasks. The smaller a single step is, the more the transaction cost will be. And vice-versa, the larger the range is, the longer you need to wait the higher ranges tasks to finish: the total execution time is the longest task's.


This project comprises examples of the following techniques:

  • dynamic dialog creation on RunBaseBatch
  • wrapping for execution in CIL
  • execution time calc
  • multiple batch task creation
  • try-catch exception handling for concurrent table updating
  • InteropPermission assertion


Happy AX mining!