This post is part of our series on Python. Part 1 came in three parts: a, b, and c. We also have a post on setting up python. And on how to write a python job submission script. For all python fun, search “python” in the search box on the right.
Transcript
At yesterday’s group meeting, I promised a transcript of the live demo. I’m pasting it below with light edits for formatting.
1 import pandas 2 import matplotlib 3 metrics = pandas.read_table("metrics.txt") 4 metrics 5 metrics[0:40] 6 metrics["roundeddown"] = 3000 * ( metrics["NFE"] // 3000) 7 metrics[0:40] 8 grouped = metrics.groupby(["model", "roundeddown"]) 9 mean = grouped.mean() 10 mean 11 mean.ix["response"] 12 grouped = grouped["SBX+PM"] 13 mean = grouped.mean() 14 mean 15 fig = pyplot.figure(figsize=(10,6)) 16 ax = fig.add_subplot(1,1,1) 17 ax.plot(mean.ix["response"].index.values, 18 mean.ix["response"].values, color = 'r') 19 fig 20 ax.plot(mean.ix["gasp"].index.values, 21 mean.ix["gasp"].values, color = 'b') 22 fig 23 summary = pandas.DataFrame( 24 data = { 25 "mean":grouped.mean(), 26 "tenth":grouped.quantile(0.1), 27 "ninetieth":grouped.quantile(0.9) 28 } 29 ) 30 summary 31 fig = pyplot.figure(figsize=(10,12)) 32 ax = fig.add_subplot(2,1,1) 33 index = summary.ix["gasp"].index.values 34 mean = summary.ix["gasp"]["mean"].values 35 tenth = summary.ix["gasp"]["tenth"].values 36 ninetieth=summary.ix["gasp"]["ninetieth"].values 37 ax.plot(index, mean, 38 index, tenth, 39 index, ninetieth, 40 color='k') 41 fig 42 ax.fill_between(index,tenth,ninetieth,color='k',alpha=0.01) 43 fig 44 ax.fill_between(index,tenth,ninetieth,color='k',alpha=0.1) 45 fig 46 ax = fig.add_subplot(2,1,2) 47 index = summary.ix["response"].index.values 48 mean = summary.ix["response"]["mean"].values 49 tenth = summary.ix["response"]["tenth"].values 50 ninetieth=summary.ix["response"]["ninetieth"].values 51 ax.plot(index, mean, 52 index, tenth, 53 index, ninetieth, 54 color='k') 55 fig 56 ax.fill_between(index,tenth,ninetieth,color='k',alpha=0.1) 57 fig 58 for ax in fig.axes: 59 ax.set_yscale("log") 60 ax.set_ylim(0.0001, 1.0) 61 ax.set_xlim(0,103000) 62 fig
Description
The transcript picks up where your homework left off. We have 50 seeds worth of data for two models:
And we want to produce a summary plot:
You should be able to use text-based ipython on the cluster without any additional setup. However, you’ll need to add from matplotlib import pyplot to the top before you start typing in code from the transcript. (The HTML notebook I used for the demo imports pyplot automatically.)
If you want to see your figures, you’ll have to take the somewhat cumbersome steps of saving them (fig.savefig("filename")), copying them to your local machine (scp user1234@hammer.rcc.psu.edu:workingdirectory/filename.png .), and then viewing them.
I used an ipython HTML notebook for the demo. I’ll be making two posts about how to do this: one about how I did it for the demo, and another about installing ipython on your desktop.
Pingback: Python Data Analysis Part 1a: Borg Runtime Metrics Plots (Preparing the Data) | Pat Reed Group Research Tips Blog
Pingback: Python Data Analysis Part 1c: Borg Runtime Metrics Plots | Pat Reed Group Research Tips Blog
Pingback: Re-evaluating solutions using Python subprocesses | Pat Reed Group Research Tips Blog
Pingback: Python for automating cluster tasks: Part 1, Getting started | Water Programming: A Collaborative Research Blog
Pingback: Python for automating cluster tasks: Part 2, More advanced commands | Water Programming: A Collaborative Research Blog
Pingback: Water Programming Blog Guide (Part I) – Water Programming: A Collaborative Research Blog