A colleague, Mahlon Anderson, and I were thinking of ways of splitting up a fairly large XML file, which had approximately 27K records in it. I wanted to split this file into smaller ones, each having about 250-300 records, because my former web host service kept complaining about constant CPU quota overloads during uploads. A Perl based splitter script quickly came to mind.
With the web host service, I had plenty disk space and plenty bandwidth but limited CPU usage. Apparently, I didn’t notice that sticky point in the fine print while signing up for the service.
A different splitting implementation was later used as the permanent solution, but here is Mahlon's "quick and dirty" XML Perl splitter-- printed with his permission of course.
# vi split.pl
#!/usr/bin/perl
$file = @ARGV[0];
open(_FH, "< $file") or die "Unable to open file\n";
$count = 0;
$files_counter=1;
$max_records = 300;
while (_FH)
{
if($count == 0)
{
$filename = $file . "_part_" . $files_counter;
open(FH2, "> $filename") or die "Unable to open file: $filename\n";
$count++;
}
if (grep /<\/item>/, $_ )
{
$count++;
}
print FH2 $_;
if ($count == $max_records + 1)
{
$count = 0;
$files_counter++;
close(FH2);
}
}
:wq!
# ./split.pl bigxmlfile.xml