Thank you for all your replies. This seems to get pretty well most sites I want (search based sites I can't but oh well, my employeer will just need to deal with it)
One problem I seem to have though, is http://www.agrium.com/products_services/msds/ site.
There are 3 sections, Dry products, liquid products, specialty
products. I want to download all pdfs on each on of these sections.
The problem I am having is wget doesn't seem to crawl into those links
no matter what I try. I am guessing it is most likely cause it is a JSP
page.
SO for example, I was wanting it to crawl to dry section, and then
download all the PDFs on that page, then move to the liquid section and
do the same.
Now I could try linking to the page directly, but I need all 3 pages and want to do it in 1 command. I know I could include the three separate links but was wanting it to crawl to each one.
Wget seems to work with ASP pages, however I have to seem to put -A *.asp,*.pdf . If i don't download the asp pages it wont crawl the links on it for some reason.
But thanks again for suggesting this wget. it saving me a bunch of headaches.