Parallel Items/Product Validating or Deleting

If you need to validate or delete items/products or any other records in a BIG number, it is better to run such processing, first, in CIL, second in parallel threads.

This project is to demonstrate this approach.

The whole concept is similar to what I explained in one of my previous blogpost about Multi thread parallelism and a dispatching table for finding a minimum

myInventItemProcessBatch class populates a special table containing RecIds to be processed and thread number they belong to.

Based on the user selection, it creates appropriate number of batch tasks that can run independently with their progress percentage.

Feel free to elaborate this project by adding new types of processing or new table to process. Also it is probably a good idea to add a new column to the table to separate different instances myInventItemProcessBatch simultaneously running in the same environment.


myInventItemProcessBatch 

private static server int64 populateItems2Process(str 20 _what2find, int _batchThreads)
{
    myInventItemProcessTable    myInventItemProcessTable;
    InventTable                 inventTable;
    int                         firstThread = 1;
    Counter                     countr;
    // flush all previously created items from the table
    delete_from myInventItemProcessTable;
    // insert all needed items in one shot. this part can be refactored to use Query instead
    insert_recordset myInventItemProcessTable (threadNum, ItemRecId, ItemId)
    select firstThread, RecId, ItemId from InventTable
    where
        inventTable.itemId like _what2find;
    // now group them in threads by simply enumerating them from 1 to N
    countr=1;
    ttsBegin;
    while select forUpdate myInventItemProcessTable
    {
        myInventItemProcessTable.threadNum = countr;
        myInventItemProcessTable.update();

        countr++;

        if(countr > _batchThreads)
        {
            countr=1;
        }
    }
    ttsCommit;
    // return the total number of items to process
    select count(RecId) from myInventItemProcessTable;

    return myInventItemProcessTable.RecId;
}

public void run()
{
    // get all required items by their RecIds in the table and group them in threads
    int64 totalRecords = myInventItemProcessBatch::populateItems2Process(what2find, batchThreads);
    if(totalRecords)
    {
        info(strFmt("Found %1 items like '%2' to %3", totalRecords, what2find, processType));
        // create number of batch tasks to parallel processing
        this.scheduleBatchJobs();
    }
    else
    {
        warning(strFmt("There are no items like '%1'", what2find));
    }
}

myInventItemProcessTask process()

... 

select count(RecId) from inventTable
        exists join myInventItemProcessTable
        where
            myInventItemProcessTable.ItemRecId == inventTable.RecId &&
            myInventItemProcessTable.threadNum == threadNum;
    // total number of lines to be processed
    totalLines = inventTable.reciD;
    // to enjoy our bored user during a few next hours
    // this progress just updates percentage in Batch task form
    progressServer = RunbaseProgress::newServerProgress(1, newGuid(), -1, DateTimeUtil::minValue());
    progressServer.setTotal(totalLines);

    while select inventTable
        exists join myInventItemProcessTable
        where
            myInventItemProcessTable.ItemRecId == inventTable.RecId &&
            myInventItemProcessTable.threadNum == threadNum
    {
        progressServer.incCount();

        try
        {
            // RUN YUR LOGIC HERE //////////////////////

...

Multi thread parallelism and a dispatching table for finding a minimum

In my free time I enjoy by solving programming puzzles from Advent of Code website. Some of them are pretty simple, though others could be tricky, however, all of them are always witty. Of course, I do it in AX so that I could use as much its power as possible.

The day 4 Ideal Stocking Stuffer became a die-hard to me. And it is not because of its "business complexity" -- you simply need to find the lowest positive number producing an MD5 hash for a given secret code, so that such a hash, in hexadecimal, starts with at least five zeroes.

Honestly, I have a vague idea about MD5 hash math -- I just took a working example and injected it into my class.

The stumbling point here was calculation time. Even for the first part of the puzzle, which is always easier than than the second one, it took so much time that I started flirting with the idea to improve performance.

Wrapping the MD5 hash calculation method so that it could be run in CIL got it faster but not enough to be happy.

The next idea was batch task execution in parallel threads, like it is brilliantly explained by Ganas1 in four chapter blog series:

Batch Bundling

Individual Task Modeling

Top Picking

Comparison of the three techniques

However, we need to find the lowest positive number; therefore, we do not know how many tasks must be created. (Let's assume that we are limited with the maximum of Int64)


My solution is the following.

I created a table, which is to centrally dispatch creating, executing, and stopping batch tasks based on a separate, sequentially assigned positive number ranges. So, for each batch task it keeps the assigned thread number, ranges, execution status and found results, if any.

The batch task generating class creates them for a given number of logical processors, four in my environment.

Each task checks the table for a found result. If it is already found in any range, it stops.
If not, it looks for the highest range from the table and than tries to add a new record. In case of success, it runs finding the lowest number in the given range.

If such a number is found in the current thread, this value becomes a new candidate only if there are no results found in the lower ranges and no any lower ranges still running.

Now blood runs faster: even the second part of the job did not give me a pause to get another beer from the fridge.

However, it is up to your judgement to set up the right range size and number of parallel tasks. The smaller a single step is, the more the transaction cost will be. And vice-versa, the larger the range is, the longer you need to wait the higher ranges tasks to finish: the total execution time is the longest task's.

This project comprises examples of the following techniques:

• dynamic dialog creation on RunBaseBatch
• wrapping for execution in CIL
• execution time calc
• multiple batch task creation
• try-catch exception handling for concurrent table updating
• InteropPermission assertion

Happy AX mining!