5

remove lines with duplicate first field

view full story
linux-howto

http://www.unix.com – Trying to cut down the size of some log files. Now that I write this out it looks more dificult than i thought it would be. Need a bash script or command that goes sequentially through all lines of a file, and does this: if field1 (space separated) is the number 2012 print the entire line. Do this DEFINITELY ALWAYS. if field1 is not the number 2012, follow this rule: if field1 of current line is same as field1 of previous line, DONT print the line, otherwise DO print the line. Another way of saying the rule is: only if field1 of current line is DIFFERENT than field1 of the (HowTos)

jiqi2's picture
Submitted by jiqi2 on 03/18/2012 – Made popular on 03/18/2012

Stories similar to remove lines with duplicate first field

This is the problem: Write a script that will make a backup of a file giving it a ‘.bak’ extension & verify that it works. I have tried a number of different scripts that haven't worked and I haven't seen anything really concise and to the point via google.

Column manipulation 37 weeks 1 day ago

Hi,

I have an input file in which I need to add a column with the value of part of another column.

I.e.

Hi All,

I am trying to generate a file through a LINUX scripting as shown below

export field1=`cat dir/filename1.txt |wc -l` echo "field1 : $field1 >>dir/WeeklySummaryReport.txt export field2=`cat dir/filename2.txt |wc -l` echo "field2 : $field2 >>dir/WeeklySummaryReport.txt

While checking the file, i could see the output is coming as a single line in the file.

Using awk to mix two files 8 weeks 6 days ago

I have two tab delimited files

file1.txt

field1 field2 field3

file2.txt

field1 value f11 field1 value f12 field1 value f13 field2 value f21 field2 value f22 field2 value f23 field3 value f31

I want the output

field1 value f11, value f12 , valuef13 field2 value f21, value f22 , valuef23 field3 value f31

I am trying this

awk -F"\t" 'NR==FNR{a[$1] = $1;next} { print a[$

Hi, I have an input file with below data and rules file to apply search and replace by each field in the input based on exact value or pattern. Could you please help me with unix script to read input file and rules file and then create the output and reject files based on the rules file. Input file: field1 field2 rc11 rc12 rc$21 rc#22 XX31 yy32 rc41 r!42

rules file: field search c

I'm looking to obtain two fields from a string. Here's the example:

$_LANG['field1'] = "field two";

I want to be able to return: field1 fieldtwo

I have a file and some records may contain double quotes at beginning and at end of line. So how do I strip them?

For Example, file is somethings like this

Code: Field1;Field2;Field3 01;'Test';'Test Field3' "01;'This is 2nd field';This is 3rd field' " Desired Output is:

Code: Field1;Field2;Field3 01;'Test';'Test Field3' 01;'This is 2nd field';This is 3rd field' Thanks

I have a file and some records may contain double quotes at beginning and at end of line. So how do I strip them?

For Example, file is somethings like this

Code: Field1;Field2;Field3 01;'Test';'Test Field3' "01;'This is 2nd field';This is 3rd field' " Desired Output is:

Code: Field1;Field2;Field3 01;'Test';'Test Field3' 01;'This is 2nd field';This is 3rd field' Thanks

I have the file like this

field1****This is, sample text ", here and there"@@@@field2****This is, sample text ", here and there"@@@@field3****This is, sample text ", here and there"@@@@

**** is the field separator

@@@@ is the record separator

i want the csv with like this

field1, "This is, sample text ", here and there""

so that i can open in microsoft excel and its displays in two columns