[Data-Mongers] web access tools

[Arthur Newhall]

Hi Nan,

     I think using python or perl would be the most useful and easiest since
  all your functions/utilities are already included with the languages  and
should be machine independent.  I have done something like this in perl
a while ago but will have to dig it out.

There are some utilities in 
ftp://acoustics.whoi.edu/pub/perl_contrib_whoi that
Steve Lerner and I put together, awhile ago as well (2007), but might might
be useful to look at.  We were trying to start up a WHOI wiki-like contrib
for matlab/python/perl/... scripts.  Perhaps we should think about it again.

Arthur



     On 04/05/2013 10:34 AM, Nan Galbraith wrote:
>   
>
> A project I'm working on needs to pull resources from a lot
> of different web servers at various agencies and research centers.
> These will initially be images, but may include data files and
> mixed format metadata later. These downloads will be done
> on different schedules, and need to be automated - they'll be
> run via cron.
>
> Some downloads will need to be set up to generate a custom
> URL, for sites where the resource name changes (e.g. to include
> a date or some other index) and each download will be followed
> by some post-processing to either modify images, add them
> to time series, shuffle them into a file structure, or ... something
> else, TBD.
>
> When I've set up similar projects in the past, I've used wget or
> lynx - they're easy to set up to be called from a shell script and
> they send the output wherever you want.  This time I'm thinking
> of using python; I think it handles redirects and logins (which
> may be required at some of my sites) more smoothly than the
> others.
>
> I just wanted to ask if anyone has a different solution for
> this kind of project. It will be running under Mac OS X 10.8.
>