Problem description & analysis

Below is data in text file txt.txt:

Each row in the text file contains a number. There are duplicates and rows are already sorted. We are trying to trim the file to remove the duplicates and generate a new ordered file, as shown below:

Solution

Write the following script p1.dfx in esProc:

Explanation:

A1   Import the txt data; @i option enables returning a sequence when the result set has only one column.

A2  Perform distinct on A1’s ordered sequence and export result to result.txt.

Explanation:

A1   Import the txt data; @i option enables returning a sequence when the result set has only one column.

A2  Perform distinct on A1’s ordered sequence and export result to result.txt.

SPL open source address

Download