Deleting specific lines and duplicates from a 11gb wordlist text file

I have a file called abc. The content of abc is:

ccc abc ccc ccc a b dd ccc

I want to sort the lines of the file and delete all duplicates (in this case ccc are duplicates).

In the shell script I use this:

sort -u < $1

But the sorted result becomes the standard output instead of saved into the abc file. How do I do this?

A huge (up to 2 GiB) text file of mine contains about 100 exact duplicates of every line in it (useless in my case, as the file is a CSV-like data table).

What I need is to remove all the repetitions while (preferably, but this can be sacrificed for a significant performance boost) maintaining the original sequence order. In the result each line is to be unique.

Quote:

Hi Am using unix ksh..

I have a file called FILE

Code: cat FILE 11/11/2012 11/11/2012 12/11/2012 15/11/2012 need to remove the duplicates dates ( ie 11/11/2012 is present two times i need remove one duplicates date )

Code: Need outputs like this 11/11/2012 12/11/2012 15/11/2012 I have tried using awk command and uniq command i

Hi All,

Need a small help in writing a shell script which can delete a few lines from a file which is currently being used by another process.

File gets appended using tee -a command due to which its size is getting increased. Contents like :

Code: [INFO] 25/09/2012 05:18 Run ID:56579677-1 My requirement is to remove lines which are more than 1 month old...

Hello to all backtrack comunity.

I've downloaded a wordlist that was inside 4 .rar files, after i unpacked the wordlist it has 7Gb, since aircrack-ng only allows 2Gb wordlists i would like anyone to help me on how to split the word list like in 4 parts or so... I've tried Aircrack's forum but nothing so far.

Thank you

Does the "64bit backtrack r3 5" supports larger wordlist than 2gb? (32bit variation does not support larger wordlist than 2gb) or do i have to split my wordlist again. (15gb wordlist)

Hello All,

This is not an Ubuntu question, more like a command question but since I use Ubuntu I just jumped into this section.

I am trying to sort a text file and remove all the duplicates. Every single line within this file looks like this:

I have a file with many rows, and each row has a timestamp at the starting, like

[Thread-3] (21/09/12 06:17:38:672) logged message from code.....

So, I frequently check 2 things from this log file.

First few rows, that has the global conditions and start time is also given. Last few rows, that has the exit status with some other info.

Is there any quick handy single command that could let m

Hi there.

I've been using the aircrack-ng suite to test my wireless network. Using the smaller wordlist's from g0tmi1k i wasn't able to do it, however i came accross a more sophisticated attack since it tests not only alot more passphrase's (aircrack-ng limits the wordlist to 2GB) it also does it a lot faster.