1

Extracting lines by key from very large file

view story
linux-howto

http://unix.stackexchange.com – I have a 42M line text file. Th first nine characters of each line are a numeric key. What is the most efficient way to extract only the lines whose key exists in another list of about 1.5M keys? Both the file and the list of keys are sorted. (HowTos)

missing's picture
Submitted by missing on 08/22/2012

Stories similar to Extracting lines by key from very large file

Hi,

I am trying to extract lines from a text file given a text file containing line numbers to be extracted from the first file. How do I go about doing this? Thanks!

I have a large text file containing over 180k lines and another text file containing about 1k. I would like to remove lines in the 180k-line file that exist in the 1k-line file. I thought there was a simple way to do this but I haven't come across it yet. Any advice? Thanks :)

I've been a Unix admin for nearly 30 years and never learned AWK. I've seen several similar posts here, but haven't been able to adapt the answers to my situation. AWK is so damn cryptic! ;)

I have a single file with ~900 lines (CSV list). Each line starts with an ID, but with different stuff after it which I want to merge onto a single line.

Linux's head selects a number of lines from the top of a file.

I have a file like the following:

a,b,c,d,e,f 1,2,3,4,5,6 7,8,9,0,1,2

I need to join the last two lines with a comma, like so:

a,b,c,d,e,f 1,2,3,4,5,6,7,8,9,0,1,2

The problem is that this is all inside a loop that will add two new lines to that file, say

a,b,c,d,e,f 1,2,3,4,5,6,7,8,9,0,1,2 0,9,8,7,6,5 4,3,2,1,0,9

And now I need to do the same to that last line for the file to end like

a,

Hello,

I was wondering how is it possible if I use this command:

Code: awk 'NR >= 998 && NR <= 1000' file.txt To exit after parsing the 1000th line ( last line targeted) ???

I have an application that is logging to a plain text log file (myapp.log) but it doesn't seem to be writing new line characters at the end of each log entry. If I execute a command like tail -n 50 myapp.log I actually receive hundreds of "lines" of text (log entries).

This log file is very large, roughly 1GB, who knows how long ago the last CR and/or LF was inserterd.

How can i insert text at a particular line number or before/after a search pattern? The text i want to insert is of 8-10 lines including new line characters.

In the same spirit as this other question: cat line X to line Y on a huge file:

Is there a way to open from within Emacs (and show on a buffer) a given set of lines (e.g. all lines between line X and Y) from a huge text file?

E.g.