Saturday, February 10, 2018

How to update progress in DMF batch tasks

I do not  know why but standard DMF writer does not update its batch task progress. It is really dull to run DMF target step and see 0% progress for long time trying to guess only what ETA is.

Fortunately it is easy to fix. The whole idea is simple: init a server side progress and update it every time the next record is processed.


DMFEntityWriter class
private void myProgressServerInit(RefRecId _startRecId, RefRecId _endRecId)
{
    if(xSession::isCLRSession())
    {
        // this progress just updates percentage in Batch task form
        myProgressServer = RunbaseProgress::newServerProgress(1, newGuid(), -1, DateTimeUtil::minValue());
        myProgressServer.setTotal(_endRecId - _startRecId);
    }
}





public container write(DMFDefinitionGroupExecution  _definitionGroupExecution,
                       DMFdefinationGroupName       _definitionGroup,
                       DMFExecutionID               _executionId,
                       DMFEntity                    _entity,
                       boolean                      _onlyErrored,
                       boolean                      _onlySelected,
                       RefRecId                     _startRefRecId = 0,
                       RefRecId                     _endRefRecId = 0,
                       boolean                      _isCompare = false,
                       DmfStagingBundleId           _bundleId = 0)
{
    Common                      target;
 ...

    if (_entity.TargetIsSetBased)
    {
        ...
    }
    else
    {
        ...

        while (nextStartRecId <= lastRecId)
        {
            ...

            try
            {
                
                this.myProgressServerInit(startRefRecId, endRefRecId);
                
                ...
                while select staging where staging.(defGroupFieldId) == _definitionGroup && staging.(execFieldId) == _executionId
                            && ( (!_onlyErrored && !_onlySelected) ||
                                    (_onlyErrored && staging.(transferStatusFieldId) == DMFTransferStatus::Error) ||
                                    (_onlySelected && staging.(selectedFieldId) == NoYes::Yes)
                                )
                    join tmpStagingRecords where
                        tmpStagingRecords.RecId                 >= startRefRecId   &&
                        tmpStagingRecords.RecId                 <= endRefRecId     &&
                        tmpStagingRecords.StagingRecordRecId    == staging.RecId
                {
                    try
                    {
                        
                        this.myProgressServerIncCount();
                        

                        ...
     }
     catch
     {
     ...
     }
            }
        }
        
        this.myProgressServerKill();
        
    }
 ...
    return [newCount, updCount, stagingLogRecId,msgDisplayed];
}

Why my batch job is still waiting?

You probably have compilation errors in CIL. Try to recompile full CIL.

Tuesday, February 6, 2018

Why your method is still not running in CIL

Please do not forget about the third parameter when you call your CIL wrapper.




It won't run in CIL if the current transaction level is not equal to zero, like in the following example with the standard import of Zip codes.



However, you can try to force ignoring TTS level at your own risk.


It works perfectly for my scenario.


This is how to check if your code is running in CIL.

Friday, February 2, 2018

Parallel Items/Product Validating or Deleting

If you need to validate or delete items/products or any other records in a BIG number, it is better to run such processing, first, in CIL, second in parallel threads.

This project is to demonstrate this approach.

The whole concept is similar to what I explained in one of my previous blogpost about Multi thread parallelism and a dispatching table for finding a minimum

myInventItemProcessBatch class populates a special table containing RecIds to be processed and thread number they belong to.



Based on the user selection, it creates appropriate number of batch tasks that can run independently with their progress percentage.




Feel free to elaborate this project by adding new types of processing or new table to process. Also it is probably a good idea to add a new column to the table to separate different instances myInventItemProcessBatch simultaneously running in the same environment.


myInventItemProcessBatch 

private static server int64 populateItems2Process(str 20 _what2find, int _batchThreads)
{
    myInventItemProcessTable    myInventItemProcessTable;
    InventTable                 inventTable;
    int                         firstThread = 1;
    Counter                     countr;
    // flush all previously created items from the table
    delete_from myInventItemProcessTable;
    // insert all needed items in one shot. this part can be refactored to use Query instead
    insert_recordset myInventItemProcessTable (threadNum, ItemRecId, ItemId)
    select firstThread, RecId, ItemId from InventTable
    where
        inventTable.itemId like _what2find;
    // now group them in threads by simply enumerating them from 1 to N
    countr=1;
    ttsBegin;
    while select forUpdate myInventItemProcessTable
    {
        myInventItemProcessTable.threadNum = countr;
        myInventItemProcessTable.update();

        countr++;

        if(countr > _batchThreads)
        {
            countr=1;
        }
    }
    ttsCommit;
    // return the total number of items to process
    select count(RecId) from myInventItemProcessTable;

    return myInventItemProcessTable.RecId;
}

public void run()
{
    // get all required items by their RecIds in the table and group them in threads
    int64 totalRecords = myInventItemProcessBatch::populateItems2Process(what2find, batchThreads);
    if(totalRecords)
    {
        info(strFmt("Found %1 items like '%2' to %3", totalRecords, what2find, processType));
        // create number of batch tasks to parallel processing
        this.scheduleBatchJobs();
    }
    else
    {
        warning(strFmt("There are no items like '%1'", what2find));
    }
}
myInventItemProcessTask process()

... 
select count(RecId) from inventTable
        exists join myInventItemProcessTable
        where
            myInventItemProcessTable.ItemRecId == inventTable.RecId &&
            myInventItemProcessTable.threadNum == threadNum;
    // total number of lines to be processed
    totalLines = inventTable.reciD;
    // to enjoy our bored user during a few next hours
    // this progress just updates percentage in Batch task form
    progressServer = RunbaseProgress::newServerProgress(1, newGuid(), -1, DateTimeUtil::minValue());
    progressServer.setTotal(totalLines);

    while select inventTable
        exists join myInventItemProcessTable
        where
            myInventItemProcessTable.ItemRecId == inventTable.RecId &&
            myInventItemProcessTable.threadNum == threadNum
    {
        progressServer.incCount();

        try
        {
            // RUN YUR LOGIC HERE //////////////////////
...