Standard Import Procedure

The complete framework for standard imports is predetermined. The structure of the import file and the mapping of the fields to be imported to the DOKA fields is done via a rule 'DoxxxMap' on level 5 and is individually customizable. The fields of the import file have to be read with the function “GetNextField()”.

The import file has to be available in a text format and the fields have to be available in full length. Text-delimited format are not supported by DOKA.

If the field mappings for a customer are used in several transactions, the actual mapping rule is outsourced into a module and will be called from 'DoxxxMap'.

The module IMPMOD provides all standard functions. The parts designed just for the current import are located in the main transaction.

The standard functions are:

• Defaulting and checking the import file
• Frame logic for processing the files
• Creating the log
• Archiving the import file

Furthermore, IMPMOD provides the subpanel for releasing the import file.

Log file:

At the end of the process a list with the numbers of processed records is displayed:

• Read Records
• Imported Records
• Updated Records
• Unaltered Records
• Erroneous Records

The Batch Importer is a standard module that executes the generic parts of an import.

The organizational frame of the DOKA-NG MyModelbank is:

1. BATIN (for Batch - Input) as subdirectory in the DATA partition as a common directory, in which the files to be imported are stored and out of which files are also imported, respectively.
2. Successfully imported files are moved to a subdirectory ARC\<yyyymm>.
3. Files that cannot be imported are moved into a subdirectory ERROR.
4. Optionally, date and time of the import can be added between filenames and extensions.
5. Each import is logged in a process log created via LOGMOD. The log also contains the file names that have been created when moving the files.
6. The importers can also be started extraneous via FileTransfer script, that can be executed directly after completing the process. Thus, it must be possible to pass the file name to be imported via command line.
7. The various import transactions can all be identified via a unitary prefix (IMPxxx).

IMPMOD is a vital module for the generic functions of all importers.

The module uses the following interfaces:

• Field with path name where the file to be imported is located
• Field with file name of the file to be imported
• A set of provided routines that are processed in the import file

StartOfFile
ImportLine
EndOfFile

The field mapping of the individual fields is done in the routine 'ImportLine'. The mapping should be supported by the module in this respect, that, depending on the structure, whether the respective positions per target field are cut out via MID or that the res. next field content can easily be called in CSV structures.

'StartOfFile' is merely used for initializing control or group changing routines.

Subsequently, 'EndOfFile' can end “delayedWrite” operations.

All three routines can report errors to IMPMOD, in order that IMPMOD can abort the import in that case.

Fixed file names, prefixes or the like can easily be implemented into the importer transaction, if a customers requests these features.

In DOKA 4 the file formats of the import files were based on fixed record length (COBOL-FD). In DOKA-NG the file formats are based on text files (i.e. with CR/LF or LF delimiters) in order that TradeDesign can process the files correctly (in TradeDesign Streamload and line-by-line processing).

Currently, the problem is that in the fixed record structures are embedded compressed fields to some extent. If the compressed value is “0”, these fields contain a “0C = Formfeed”. Furthermore, the zeros are displayed as “Hex-00” in values with several digits, which is also a problem for TradeDesign. In order to solve these problems, the module SYSTDT can be used to convert files into line with a fixed record length. Compressed numbers that are also included in the text can be transferred to another processable format. Subsequently, the file is directly accessible in TradeDesign line-by-line as a stream.

On the Client/Server level the import file can be subdivided into text lines via the 'dd' tool, if required. In doing so, however, the possibly embedded Hex-00 will not be converted.

Procedure:

“dd if=P991001.DAT of=P991001.TXT cbs=120 conv=unblock”

'if' represents 'InputFile'
'of' represents 'OutputFile'
'cbs' represents 'ConversionBlockSize'
'conv=block / unblock' represents the direction of the conversion

Specific areas of the input file can be cut out with s“eek=RecordNumber”, “skip=SkipInputBlocks”, and “count=InputBlocks”. After converting the fixed format into a text format, this format can be processed in TradeDesign line by line. Additionally, the import file can also be read via normal methods (e.g. it can be displayed with SYSIXF).

Like in the whole application, the central decoding module SYSTDT (see 'SYSTDT - External File Conversion Routines') is used in the import procedures, in order to use all available decodings.

Note:
An already decoded line is passed to an import transaction. In case the original data are required (i.e. the data, that have not been decoded via SYSTDT),  “GetOriginalLine” needs to be called. This can be necessary, if the compressed fields have to be read out and handled. The compressed fields may NOT be decoded.

Under the following link details about the Implementation can be found.