[Data-Mongers] web access tools

[Andrew Maffei]

Nan,

Massimo is an RPI staffmember/WHOI guest investigator working on the project here in WH, working w/for Peter Fox. You can also talk to Stace Beaulieu, she is leading from the WHOI side. 

Among other things iPython notebook can integrate Python, MATLAB, R, Latex and a few other languages as notebook "cells" that flow together. Info can be easily passed from one cell to another. We have the experience of directly copying both existing MATLAB, R, Python, LaTex code into adjacent cells, and building a flow from source data access to paper publication.

I just spoke to Massimo and he offered these 2 links as intros to iPython for those interested:

https://www.youtube.com/watch?v=F4rFuIb1Ie4
http://vimeo.com/53094837

--Andy

On Apr 29, 2013, at 7:39 AM, Cyndy Chandler <cchandler at whoi.edu> wrote:

> Nan,
> 
> did you hear back from anyone working on the ECO-OP project? Massimo Di 
> Stefano (working with Andy Maffei and others in the Oceans Informatics 
> Group) has done some really interesting work using iPython to 
> essentially track workflow, and doing many of the same activities you 
> mention for your project.  It would be worth getting in touch with him 
> to see if he has any suggestions.
> 
> https://www.whoi.edu/more.go?username=mdistefano
> 
> ECO-OP project
> http://tw.rpi.edu/web/project/ECOOP
> 
> Cheers, Cyndy
> 
> 
> On 4/5/2013 10:34 AM, Nan Galbraith wrote:
>> Hi all -
>> 
>> I'm not sure which (if either) of these lists is active, so I'm
>> trying both - I hope mailman takes care of duplicates, but
>> if not, my apologies for cross posting.
>> 
>> A project I'm working on needs to pull resources from a lot
>> of different web servers at various agencies and research centers.
>> These will initially be images, but may include data files and
>> mixed format metadata later. These downloads will be done
>> on different schedules, and need to be automated - they'll be
>> run via cron.
>> 
>> Some downloads will need to be set up to generate a custom
>> URL, for sites where the resource name changes (e.g. to include
>> a date or some other index) and each download will be followed
>> by some post-processing to either modify images, add them
>> to time series, shuffle them into a file structure, or ... something
>> else, TBD.
>> 
>> When I've set up similar projects in the past, I've used wget or
>> lynx - they're easy to set up to be called from a shell script and
>> they send the output wherever you want.  This time I'm thinking
>> of using python; I think it handles redirects and logins (which
>> may be required at some of my sites) more smoothly than the
>> others.
>> 
>> I just wanted to ask if anyone has a different solution for
>> this kind of project. It will be running under Mac OS X 10.8.
>> 
>> Thanks in advance for any suggestions -
>> Nan
>> 
>> 
> 
> -- 
> Cyndy Chandler          | voice: (508) 289-2765
> MS #36, Shiverick, WHOI | Office hrs: M-F ~ 7-6
> Woods Hole, MA  02543   | FAX: (508) 289-2009
> cchandler at whoi.edu      | Skype: cyndy.chandler
> http://www.whoi.edu/profile/cchandler
> Biological and Chemical Oceanography Data Management Office
> Department of Marine Chemistry and Geochemistry
> Woods Hole Oceanographic Institution
> 
> _______________________________________________
> Data-Mongers maillist  -  Data-Mongers at whoi.edu
> http://mailman.whoi.edu/mailman/listinfo/data-mongers
>