Unix Split, Join & Validate Files

| Posted by watashii | Filed under Programming, Unix

Problem


I have a large file.

How can i split it into multiple parts, then join it back together later?

How do i make sure my files don’t corrupt?

I am using Unix.

$ ls -l *.pkg
-rw-rw-r--  1 watashii watashii 648773362 Nov 28 18:01 largefile.pkg

Splitting the files

The following Unix command splits a large file into 50MB (52,428,800 bytes) pieces.  The output is a list of files named xaa, xab etc

$ split -b 52428800 largefile.pkg
$ ls -l x*
-rw-rw-r--  1 watashii watashii 52428800 Nov 28 18:01 xaa
-rw-rw-r--  1 watashii watashii 52428800 Nov 28 18:01 xab
...
-rw-rw-r--  1 watashii watashii 19627762 Nov 28 18:01 xam

Joining the files

To join the files together, we simply use the cat command over each file in ascending order:

$ cat x* > mergedlargefile.pkg

Validating the files

To validate the files, below are some suggestions. File compare it, diff it, run a checksum to compare it.

$ ls -l *.pkg
-rw-rw-r--  1 watashii watashii 648773362 Nov 28 18:01 largefile.pkg
-rw-rw-r--  1 watashii watashii 648773362 Nov 28 18:13 mergedlargefile.pkg

$ cmp  largefile.pkg  mergedlargefile.pkg
$ cksum  largefile.pkg
3332922138  648773362  largefile.pkg
$ cksum  mergedlargefile.pkg
3332922138  648773362  mergedlargefile.pkg
$ diff  largefile.pkg  mergedlargefile.pkg

Tags: , , ,

Read more | Comments (0) | November 28th, 2008