<html><head><meta http-equiv="Content-Type" content="text/html charset=us-ascii"><meta http-equiv="Content-Type" content="text/html charset=us-ascii"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; ">Hi Nan<div><br></div><div>urllib2 should be your friend</div><div><br><div><a href="http://docs.python.org/2/howto/urllib2.html">http://docs.python.org/2/howto/urllib2.html</a></div><div><a href="http://pymotw.com/2/urllib2/">http://pymotw.com/2/urllib2/</a></div><div><br></div><div>Norman</div><div><br><div><div>On Apr 5, 2013, at 10:34 AM, Nan Galbraith <<a href="mailto:ngalbraith@whoi.edu">ngalbraith@whoi.edu</a>> wrote:</div><br class="Apple-interchange-newline"><blockquote type="cite">Hi all -<br><br>I'm not sure which (if either) of these lists is active, so I'm<br>trying both - I hope mailman takes care of duplicates, but<br>if not, my apologies for cross posting.<br><br>A project I'm working on needs to pull resources from a lot<br>of different web servers at various agencies and research centers.<br>These will initially be images, but may include data files and<br>mixed format metadata later. These downloads will be done<br>on different schedules, and need to be automated - they'll be<br>run via cron.<br><br>Some downloads will need to be set up to generate a custom<br>URL, for sites where the resource name changes (e.g. to include<br>a date or some other index) and each download will be followed<br>by some post-processing to either modify images, add them<br>to time series, shuffle them into a file structure, or ... something<br>else, TBD.<br><br>When I've set up similar projects in the past, I've used wget or<br>lynx - they're easy to set up to be called from a shell script and<br>they send the output wherever you want. This time I'm thinking<br>of using python; I think it handles redirects and logins (which<br>may be required at some of my sites) more smoothly than the<br>others.<br><br>I just wanted to ask if anyone has a different solution for<br>this kind of project. It will be running under Mac OS X 10.8.<br><br>Thanks in advance for any suggestions -<br>Nan<br><br><br>-- <br>*******************************************************<br>* Nan Galbraith (508) 289-2444 *<br>* Upper Ocean Processes Group Mail Stop 29 *<br>* Woods Hole Oceanographic Institution *<br>* Woods Hole, MA 02543 *<br>*******************************************************<br><br><br><br>_______________________________________________<br>Data-Mongers maillist - <a href="mailto:Data-Mongers@whoi.edu">Data-Mongers@whoi.edu</a><br><a href="http://mailman.whoi.edu/mailman/listinfo/data-mongers">http://mailman.whoi.edu/mailman/listinfo/data-mongers</a><br><br></blockquote></div><br></div></div></body></html>