Sunday, September 30, 2007

Download an XML feed with wget

For one of my projects, I was attempting to download a 70MB XML feed. Unfortunately, it was taking forever to download via HTTP and crashing my MSIE 6.0 and Firefox 2.0.0.7 browsers. I ended up using the wget utility to complete the task. Here's what I did.

C:\Documents and Settings\esoft\Desktop>wget www.xxxx.com/partnerfeeds/esofthub%20lists/esofthub%20xmllists.xml
--02:02:09-- http://www.xxxx.com/partnerfeeds/esofthub%20lists/esofthub%20xmllists.xml
=> `esofthub xmllists.xml'
Resolving www.xxxx.com... xx.1xx.x.2xx
Connecting to www.xxxx.com|xx.1xx.x.2xx|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 70,722,387 (67M) [text/xml]

100%[====================================>] 70,722,387 130.78K/s ETA 00:00

02:08:32 (182.75 KB/s) - `esofthub xmllists.xml' saved [70722387/70722387]

If you don't have wget, here's a couple site you can go to get it.

Download WGET for Windows
Download GNU WGET

2 comments:

mike said...

Sometimes axel is faster. '[I]t opens more than one HTTP/FTP connection per download and each connection transfers its own, separate, part of the file.'

esofthub said...

Mike,

Interesting. I'll have to try that. Thanks.

Roy