2

If I have a text file, I can sort it by command sort input.txt output.txt. Can I somehow also remove duplicate lines from the text file as well. (Ideally I would like a solution which can be done in a batch file under Windows.)

This is basically the same as asking how to remove duplicates from text file. (Since obviously I can sort first and then use a batch to remove duplicates.) The reason I have also mentioned sorting is that I assume that it is easier to remove duplicates if we know that file is sorted and they can only appear on consecutive lines. So I guess it is more likely that there is some way to remove duplicates together with sorting. (And for the purposes where I want to use this id does not matter if the order of lines in the file is changed, as long as the repeated lines are removed.)

For example from the file looking like this:

100 100 100 100 101 101 102

I would like to get

100 101 102
Improve this question
5
  • This is my first question on SU. I did my best to try to check whether this is already answered on the site and also to choose correct tags. Of course, if more experienced users of the site can suggest more suitable tags and point me out to already existing posts which I missed, I will be grateful. Commented Feb 24, 2017 at 5:44
  • Do you specifically need to keep this in native Windows tools? Is there no chance of using one of the more powerful scripting environments such as Python or NodeJS? Unfortunately, the Windows command line (and as a result, batch files) is not very efficient at processing data dynamically. It's not impossible, just very tricky. Commented Feb 24, 2017 at 5:55
  • @ZaLiTHkA Of course, if there is no solution using batch files or if it is very difficult, I will look into other possibilities. (Unless I am mistaken, unix sort can do this, so probably cygwin or gnuwin32 might probably help me.) Of course, I understand that if somebody is used to tools you mentioned, the most natural solution for such user might be to employ them. However, as the first possibility I have asked whether this is doable using batch files. Commented Feb 24, 2017 at 6:06
  • it's definitely possible, I was just wondering if there was any specific requirement for that. :) Walmart's solution handles this really well though, and far more efficiently than I would've been able to do with my knowledge of Windows CLI tools. Commented Feb 24, 2017 at 6:21
  • Please note that superuser.com is not a free script/code writing service. If you tell us what you have tried so far (include the scripts/code you are already using) and where you are stuck then we can try to help with specific problems. You should also read How do I ask a good question?. Commented Feb 24, 2017 at 11:54

1 Answer 1

Reset to default
2

Batch to sort a file with removing duplicates line?

Below is a batch script native Windows solution just as you asked for that will complete this task; just set your input and output file variables accordingly.

Bonus Note: If you change the SET InputFile=%~1 like that, then you can drag the file over to the saved batch script and it'll complete the task using the simple drag and drop method.

Batch Script

@ECHO ON SET InputFile=C:\folder\path\Input.txt ::SET InputFile=%~1 SET OutputFile=C:\folder\path\Output.txt SET PSScript=%Temp%\~tmpRemoveDupe.ps1 IF EXIST "%PSScript%" DEL /Q /F "%PSScript%" ECHO Get-Content "%InputFile%" ^| Sort-Object ^| Get-Unique ^> "%OutputFile%">>"%PSScript%" SET PowerShellDir=C:\Windows\System32\WindowsPowerShell\v1.0 CD /D "%PowerShellDir%" Powershell -ExecutionPolicy Bypass -Command "& '%PSScript%'" GOTO EOF

Further Resources

Improve this answer

You must log in to answer this question.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.