[Data-Mongers] web access tools

[Cyndy Chandler]

Nan,

did you hear back from anyone working on the ECO-OP project? Massimo Di 
Stefano (working with Andy Maffei and others in the Oceans Informatics 
Group) has done some really interesting work using iPython to 
essentially track workflow, and doing many of the same activities you 
mention for your project.  It would be worth getting in touch with him 
to see if he has any suggestions.

https://www.whoi.edu/more.go?username=mdistefano

ECO-OP project
http://tw.rpi.edu/web/project/ECOOP

Cheers, Cyndy


On 4/5/2013 10:34 AM, Nan Galbraith wrote:
> Hi all -
>
> I'm not sure which (if either) of these lists is active, so I'm
> trying both - I hope mailman takes care of duplicates, but
> if not, my apologies for cross posting.
>
> A project I'm working on needs to pull resources from a lot
> of different web servers at various agencies and research centers.
> These will initially be images, but may include data files and
> mixed format metadata later. These downloads will be done
> on different schedules, and need to be automated - they'll be
> run via cron.
>
> Some downloads will need to be set up to generate a custom
> URL, for sites where the resource name changes (e.g. to include
> a date or some other index) and each download will be followed
> by some post-processing to either modify images, add them
> to time series, shuffle them into a file structure, or ... something
> else, TBD.
>
> When I've set up similar projects in the past, I've used wget or
> lynx - they're easy to set up to be called from a shell script and
> they send the output wherever you want.  This time I'm thinking
> of using python; I think it handles redirects and logins (which
> may be required at some of my sites) more smoothly than the
> others.
>
> I just wanted to ask if anyone has a different solution for
> this kind of project. It will be running under Mac OS X 10.8.
>
> Thanks in advance for any suggestions -
> Nan
>
>

-- 
Cyndy Chandler          | voice: (508) 289-2765
MS #36, Shiverick, WHOI | Office hrs: M-F ~ 7-6
Woods Hole, MA  02543   | FAX: (508) 289-2009
cchandler at whoi.edu      | Skype: cyndy.chandler
http://www.whoi.edu/profile/cchandler
Biological and Chemical Oceanography Data Management Office
Department of Marine Chemistry and Geochemistry
Woods Hole Oceanographic Institution