The complete guide to data management. Part VI: Data Task Automation

Welcome to a new post about this guide that is coming to an end! Today we will see Data Task Automation or Data Task Automator, a functionality that helps us list and generate ordered and automated data tasks (import and export).

This functionality creates one or more data projects for us and uses packages uploaded to the LCS as a shared asset.

I leave the video and list the file structure below so you can review it.

Manifest structure:

< TestManifest name = 'Data Import DEMO v 10.0.9' >

TestManifest: is the global data of the manifest. Here we define the name in the parameter that is descriptive to recognize the manifest.

<DataFile ID='SharedSetup' name='Demo data 10.0.9 100 System and Shared'  assetType='Data package' lcsProjectId=''/>

Data File: It points us to the data packages that we are going to import from the LCS asset library. ID, is the name we give it within our manifest. Name is the name of the package as it appears in the LCS. AssetType tells us what type of asset we are going to import, in our case it will be 'DataPackage', and lcsProjectId is the LCS project in which the package is located, in our case we leave it blank indicating that it is in the asset library shared, does not belong to the particular project.

<JobDefinition ID='Impo_DEMO_HQUS'>
    <Operation>Import</Operation>
    <SkipStaging></SkipStaging>
    <Truncate></Truncate>
    <Mode>Import async</Mode>     
    <BatchFrequencyInMinutes>1</BatchFrequencyInMinutes>
    <NumberOfTimesToRunBatch >2</NumberOfTimesToRunBatch>
    <UploadFrequencyInSeconds>1</UploadFrequencyInSeconds>    
        <TotalNumberOfTimesToUploadFile>1</TotalNumberOfTimesToUploadFile>
            <SupportedDataSourceType>Package</SupportedDataSourceType>
            <ProcessMessagesInOrder>No</ProcessMessagesInOrder>
            <PreventUploadWhenZeroRecords>No</PreventUploadWhenZeroRecords>
            <UseCompanyFromMessage>Yes</UseCompanyFromMessage>
            <LegalEntity>DAT</LegalEntity>
        </JobDefinition>

JobDefinition, gives us the parameters of the data projects that will be created. The most important tags we have:

Operation: Indicates if it is import or export,

SkipStaging, if you are going to skip staging

Truncate, if you are going to truncate the data of the entities to import (that is, delete the previous information)

Mode: if the operation is of asynchronous import, or it can be a Batch, in that case the functions that relate to the recurrence of the jobs are activated

PreventUploadWhenZeroRecords, Prevents the upload of files that do not have records

LegalEntity defines the entity to process data projects

   <EntitySetup ID='Generic'><Entity name='*'>
   <SourceDataFormatName>Package</SourceDataFormatName><ChangeTracking></ChangeTracking><PublishToBYOD>
   </PublishToBYOD><DefaultRefreshType>Full push only</DefaultRefreshType>
   <ExcelWorkSheetName></ExcelWorkSheetName>
   <SelectFields>All fields</SelectFields>
   <SetBasedProcessing></SetBasedProcessing><FailBatchOnErrorForExecutionUnit>No</FailBatchOnErrorForExecutionUnit>
   <FailBatchOnErrorForLevel>No</FailBatchOnErrorForLevel>
<FailBatchOnErrorForSequence>No</FailBatchOnErrorForSequence><ParallelProcessing>
   <Threshold></Threshold>
   <TaskCount></TaskCount>
   </ParallelProcessing></Entity></EntitySetup>

Entity setup, defines how the entities contained in each package will work.

For example:

FailBatchOnError, at each level represents that if there is a failure, it cancels the processing of the packet by level, execution unit or sequence.

  < TestGroup name = 'Financial Configuration' >

Test group groups the tasks in the groups that we define for a better visualization and execution

    < TestCase Title = 'Import Shared Data'  ID = '10001' RepeatCount = '1' TraceParser ='off' TimeOut='20'>
    <DataFile RefID='SharedSetup' />
    <JobDefinition RefID='Impo_DEMO_HQUS' />
    
    <EntitySetup RefID='Generic' />

Test case defines the parameters of each operation. Like your title and your ID. It also manages the TimeOut field in minutes that it waits in case the entity is not being processed or has been stopped due to an error.

Here we must indicate which DataFile we are going to use (with its ID), also which Jobdefinition we are going to relate to it, and finally how it is going to configure the entities through the EntitySetup field. Each of these three are the ones we have previously defined, so we simply indicate the ID.

We can get hold of one of these files by checking the option to download the manifest and we will have the sample files from MS to be able to manipulate them with our definitions.

I also leave you the link of the Microsoft Learn about Data Task where we can see each field in detail, although it is in English.

I hope you can try it and see you again in another Consejo Dynamics!