Search This Blog

Loading...

Saturday, March 22, 2008

Using sed to extract lines in a text file

If you write bash scripts a lot, you are bound to run into a situation where you want to extract some lines from a file. Yesterday, I needed to extract the first line of a file, say named somefile.txt.
$ cat somefile.txt
Line 1
Line 2
Line 3
Line 4


This specific task can be easily done with this:
$ head -1 somefile.txt
Line 1


For a more complicated task, like extract the second to third lines of a file. head is inadequate.

So, let's try extracting lines using sed: the stream editor.

My first attempt uses the p sed command (for print):
$ sed 1p somefile.txt
Line 1
Line 1
Line 2
Line 3
Line 4


Note that it prints the whole file, with the first line printed twice. Why? The default output behavior is to print every line of the input file stream. The explicit 1p command just tells it to print the first line .... again.

To fix it, you need to suppress the default output (using -n), making explicit prints the only way to print to default output.
$ sed -n 1p somefile.txt
Line 1


Alternatively, you can tell sed to delete all but the first line.

$ sed '1!d' somefile.txt
Line 1


'1!d' means if a line is not(!) the first line, delete.

Note that the single quotes are necessary. Otherwise, the !d will bring back the last command you executed that starts with the letter d.


To extract a range of lines, say lines 2 to 4, you can execute either of the following:

  • $ sed -n 2,4p somefile.txt
  • $ sed '2,4!d' somefile.txt


Note that the comma specifies a range (from the line before the comma to the line after).

What if the lines you want to extract are not in sequence, say lines 1 to 2, and line 4?
$ sed -n -e 1,2p -e 4p somefile.txt
Line 1
Line 2
Line 4


If you know some different ways to extract lines in a file, please share with us by filling out a comment.

P.S. Related articles from this blog:


StumbleUpon Toolbar

48 comments:

Anonymous said...

So what if I had a huge file and I wanted to extract, say, every 4th line?

Peter Leung said...

Thanks for the question.
Please see the blog entry
http://linuxcommando.blogspot.com/2008/04/use-sed-or-perl-to-extract-every-nth.html

Chris said...

Another way to do it is with head and tail. Your way might be easier. But for example if you wanted to read ONLY line 5 from a file you could do
$ head -n 5 | tail -n 1

I'm not sure if it is more or less complicated ;)

Peter Leung said...

Interesting use of head n tail together.

Anonymous said...

...and what about if you want to extract a line ... let#s say starting with # (Komment) ?

Peter Leung said...

Try this:

sed -ne '/^#/p' somefile

mak said...

the head tail combination will not work if the file has lesser number of lines than 'n'

Anonymous said...

I want just to edit first line of file by appending something to the first line and rest is same.How should I achieve it??

Peter Leung said...

This is one way to add some text to the end of the first line of a file

sed -e '1s/$/new text/' yourfile.txt

Anonymous said...

Hi,

how can I find a certain string in an output and then print the following lines?

Usage could be xrandr to find the resolution modes of a TV:

VGA disconnected (normal left inverted right x axis y axis)
LVDS connected 1024x768+0+0 (normal left inverted right x axis y axis) 0mm x 0mm
1024x768 57.6*+ 85.0 75.0 70.1 60.0
832x624 74.6
800x600 85.1 72.2 75.0 60.3 56.2
640x480 85.0 72.8 75.0 59.9
720x400 85.0
640x400 85.1
640x350 85.1
TV connected (normal left inverted right x axis y axis)
1024x768 30.0
800x600 30.0
848x480 30.0
640x480 30.0

I would like to read all the lines after TV connected...

Thanks!

Peter Leung said...

Try this...


xrandr | sed -n '/TV connect/,$p'


which tells sed to Print only those lines within the range, defined by where it matches "TV connect" to the end of file ($).

Alan said...

Is there a way to delete line 5 if it contains some string?

Thanks.

Peter Leung said...

A quick solution using awk:

awk ' NR!=5 || !/SomeRegExpr/ {print;} ' yourfile.txt

Basically, it says, print the line if it is NOT line 5 OR (if it is line 5), it does not match Some Regular Expression.

Hope that helps.

Valentin said...

Hi Peter,

I'd like to extract values for several variables located in the file.txt on the same line:
"foo1=10 foo2==20 ... foo10=100"

in this way:

10 20 ... 100

or

10
20
...
100

For this purpose I was trying to use sed:
sed -n 's/.*foo.*=\([0-9]*\).*/\1/p' file.txt

but can got only one value: 100.

How can I got all values?

Thank you.

-Val

Peter Leung said...

Hi, Val

I believe the following will do

sed 's/foo[0-9]*=//g' file.txt

It simply removes all occurences of the "foo1=", "foo2=", ...
"foo10=" strings.

The -g means globally replace (not just the first occurence on a line by default).

(I can't verify the correctness because I am away on vacation and does not have access to a Linux computer.. sigh)

Peter

Valentin said...

Thank you, Peter!

This works!

Now I'm considering more difficult case when in the sample line present other words or numbers, like this:

"foo1=10 foo2=20 some text foo3=30 1234 foo4=40 ... foo10=100"

But the same output should be:

10 20 30 40 ... 100

Thanks,
Val

Peter Leung said...

Val,

Try this (it takes your original approach, and simplifies it somewhat).

sed 's/foo[0-9]*=\([0-9]* ?\)/\1/g' file.txt


Peter

Sundaram said...

Hi Peter,

How to extract a random line assigned by a script variable, in sed?

Peter Leung said...

sundaram

Assuming $myvar is the var

sed -n "${myvar}p" somefile.txt


Peter

Ryan said...

I've got one for you :)

I've got an html file that needs parsing. I'd like to use sed/awk/grep if possible.

I need to find a text string (xx:xx:xx) in an html file and extract the 3 lines under it and on those lines, I would like to strip off the html table tags (td).

Any ideas?
Thanks
Ryan

Ryan said...

No problem, with some time and google I got it working with a combination of things. :)

"#EXTRACT THE LINES THAT WE NEED FROM THE HTML SOURCE FILE
cat $ARCHIVELOCATION/$CURRENTDATE.html | grep -A 3 "what you are looking for" > $SCRIPTTEMP/output

#DELETE EXTRA CHARACTERS/SPACES/ETC
cat $SCRIPTTEMP/output | tr -d " tabletag," > $SCRIPTTEMP/output"

Thanks!
Ryan

Peter Leung said...

Ryan, grep -A is clever. Thanks

I come up with this awk one-liner (maybe a bit terse)

awk '/what you are looking for/ {for (i=0; i<3; i++) {getline;sub(/tabletag/,"");print}}' $ARCHIVELOCATION/$CURRENTDATE.html > $SCRIPTTEMP/output


Peter

Anonymous said...

Hi, thanks for all the tips.

I have a file with indented lines similar to this:

line 1
indent 1
indent 2
line 2
line 3
indent 1
indent 2
line 8
indent 1
line 55

I want to print lines 2 and 55 because they do not have indented comments under them. Also it would be nice to assign 2 and 55 to a variable (array).

I've googled quite a lot, but no luck yet.
Would be interesting in seeing how you guys would solve this.

Jadu Kumar Saikia said...

Peter, nice post. sed is always helpful for printing lines. Thanks.

I have some sed works done on my blog

http://unstableme.blogspot.com/search/label/Sed

// Jadu

Daya said...

how can i delete some contain using sed or grep command without using line number

Peter Leung said...

Daya

Say you want to delete all lines containing this string daya, then run this

sed '/daya/d' somefile.txt

Anonymous said...

Interesting...
I tried this on 6GB file.

[sangoh@*** transmit]$ date; perl -ne "print if($. == 551985 || $. == 552085)" multi_122708_trimmed_eur.tped >> /dev/null; date; sed -n 551985,552085p multi_122708_trimmed_eur.tped >> /dev/null; date; head -n552085 multi_122708_trimmed_eur.tped | tail -n100 >> /dev/null; date;
Tue Jan 6 00:18:54 PST 2009
Tue Jan 6 00:20:56 PST 2009
Tue Jan 6 00:22:58 PST 2009
Tue Jan 6 00:25:02 PST 2009

Louis Casillas said...

Thanks a lot! Wanted a quick solution and this was perfect!

Eri said...
This comment has been removed by the author.
Eri said...

Hi!

How can I extract randomly lines from a file?

Thanks!

Anonymous said...

Hi guys, I'm not sure if I'll get a reply from this but I wanted to give it a shot, my situation is similar to all the rest exposed before and at the same time particularly different so here it is and I hope someone could give me a hand since I simply can't figure this one out.

I have a file which contains certain blocks of text like for example the next ones
**********************************

---
---Current Info `aaaa`
---

more info
....
more info

...

more info with spaces and stuff


---
---Current Info `bbbb`
---

and again more info down here until finding again the same starting block of code



---
---Current Info `cccc`
---
***********************************


So basically what I need is a way for me to extract all the lines between the

---Current Info `aaaa`

...and

---Current Info `bbbb`

I need it to be inclusive on the top part (the aaaa line) but not necessarily on the bottom one but shouldn't matter either since I could remove X amount of lines from the bottom up easily once the real chunk of data is extracted.

I hope I was very clear and hopefully someone can give me a hand.

Thank you

Chris said...

Thanks for that post, exactly what I was looking for.

And thanks in the name of all the other commenters for all your personalized problem solving for them, Peter! This has become some sort of one-man advice forum...

Chris said...

As a reply to Eri: bash (and I don't know which other shells) can provide you with pseudo-random numbers from 0 - 32767 using $RANDOM. If you want a pseudo-random number from 0-9 say, you can use the mod operator % for example like this:

$ randnr=$((RANDOM%10))

"10" could be replaced by the number of lines in your text:

$ nrlines=`cat yourtext.txt | wc -l`

so

$ randlinenr=$((RANDOM%${nrlines}+1))

would pick a random line number of your text. Then use $randlinenr with sed

$ sed -n ${randlinenr}p yourtext.txt

Repeat until you got all you wanted.

Lastly, you give too little information, Eri, to take your question seriously. But I wanted to reply a) to help Peter and b) because I just recently had to use $RANDOM which had been totally unknown to me.

Flowerpower said...

I was wondering if you could help me on this one. I would like to search for two strings per line then print the next X integers after each string. The integers may vary in length and there is always a space between the string and integers. It should search like this for each line.

Example:
line1-sometext Lat: 36.76565 Long: -119.09011 moretext
line2-sometext Lat: 36.47777 Long: -118.0411 moretext

Desired output:
36.76565 -119.09011
36.47777 -118.0411


I hope this doesn't stump you.

Anonymous said...

Nice article as for me. It would be great to read something more about that topic. Thnx for giving this info. Linda
Kiev escort tours

Anonymous said...

Regarding extracting random lines, I like to use shuf command:
shuf file.txt | head

Andreas Papadopoulos said...

Thank you man for sharing.
I had a huge file and sed start,endp saved me.

Srikanth G K said...

Hello,
I wish to extract all lines from a text file, which contains a particular word.

like:
i wnat to print all the lines which contain "http"

How can I do this?

Peter Leung said...

Srikanth

How about this?

sed -n -e '/http/ p' yourfile

wholesale soccer Jersey said...

I wish to extract all lines from a text file, which contains a particular word.

Anonymous said...

cat file.txt | head -$desiredline | tail -1

for line 12

cat file.txt | head -12 | tail -1

Conor Smyth said...

Hi Peter,

Thanks for the original article and for still answering questions 5.5 years later!!

You've come very close to answering my question but not quite. If I want to extract certain lines you suggest using
$ sed -n -e 1,2p -e 4p somefile.txt

Now, what if I have a very large file and need to extract a subset of the lines, the index of which is too long to type manually.

Can I somehow use an index file combined with the above?

Thanks in advance and I understand if I don't hear from you, you've already gone above and beyond.

mihir mandavia said...

Hi, I want to extract lines from a file such that it extracts all the lines from line 1 to the line it encounters a string like "dump" in a new file. How can I do that ?

Thanks

Mas Ucheng 17++ said...

So what if I had a huge file and I wanted to extract, say, every 4th line?

Dana Cepat

Peter Leung said...

Hi, Dana

See this post:
http://linuxcommando.blogspot.ca/2008/04/use-sed-or-perl-to-extract-every-nth.html


Peter

Anonymous said...

I have a file of 9 lines say as below.
1
2
3
4
5
6
7
8
9


I want to 3 lines in a single line lik mentioned below
123
456
789

Can you help me with this

Nutrisi Wajib Untuk Ibu Hamil said...

great article sir

Nutrisi Wajib Untuk Ibu Hamil said...

I have a file of 9 lines say as below.
1
2
3
4
5
6
7
8
9


I want to 3 lines in a single line lik mentioned below
123
456
789

Can you help me with this