Check for duplicate lines in a variable and remove

Here is another alternative, using the power of JavaScript Arrays and Sets in JavaScript for Automation (JXA).
This is my preferred solution, and should work well even for thousands of lines.

Below is just an example written in response to your request. You will need to use as an example and/or change to meet your workflow automation needs.

Please let us know if it meets your needs.

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Example Output

MACRO:   Remove Dups from List [Example]

-~~~ VER: 2.0    2020-06-23 ~~~
Requires: KM 8.2.4+   macOS 10.11 (El Capitan)+
(Macro was written & tested using KM 9.0+ on macOS 10.14.5 (Mojave))

DOWNLOAD Macro File:

Remove Dups from List [Example].kmmacros
Note: This Macro was uploaded in a DISABLED state. You must enable before it can be triggered.