Editing a very large file can be a resource- (and time-) consuming nightmare. Having a requirement to delete the first line in such a file in-place whilst avoiding opening the file up in SomeEditor(TM) can be done in various ways, with various resource overheads.
Let me introduce you to the three methods we’ll be trying. The first uses GNU sed and the -i (inplace) option to edit the file in-place.
|
1 |
sed -i '1d' large_file |
The second methods uses perl to get the job done.
|
1 |
perl -pi -e '$_ = "" if ( $. == 1 );' large_file |
The final example uses printf (so use a shell that supports it) and ex (command-line vi).
|
1 |
printf ":1d\n:wq\n" | ex large_file |
Let’s use these methods and time them…
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 |
$ wc -l large_file 15711759 large_file $ time sed -i '1d' large_file real 3m1.870s user 2m2.406s sys 0m23.906s $ time perl -pi -e '$_ = "" if ($. == 1);' large_file real 1m46.065s user 0m0.015s sys 0m0.016s $ time printf ":1d\n:wq\n" | ex large_file real 6m53.646s user 1m52.295s sys 0m19.374s |
So the moral of this tip? Use perl for performing edits on extremely large files!