### Some notes about NumPy and SciPy Python14.12.2013

NumPy (Numerical Python) is useful for scientific computing and data analysise. Key features are writed with C so it have good performance.

Main features:

• ndarray - a powerful N-dimensional array object whis has vectorized arithmetic operations);
• set of mathematics functions for work with vectors and matrix;

Install

The following installation instruction was tested under Arch Linux.

I use Tcl/Tk for showing output from matplotlib.

```yaourt -S tk tcl
```
```mkvirtualenv -p /usr/bin/python2.7 science
```

Install NumPy, SciPy, Matplotlib, IPython

```pip install numpy, scipy, matplotlib, ipython
```

We should check backend for matplotlib and ascertain if it is tkagg.

```ipython --pylab

In : import matplotlib
In : matplotlib.matplotlib_fname()
Out : '/path/to/matplotlibrc'

grep 'backend' /path/to/matplotlibrc
backend      : tkagg
```

Basic

At the heart of NumPy is N-dimensional array (ndarray) with capability to keep big data. All items of ndarray should be the same type.

```In : from numpy import *
In : a = array([1,2,3])
In : a
Out: array([1, 2, 3])
```

Each array contains information like shape and type of elemets (dtype)

```In : a.shape
Out: (3,)
```

Creating an array from list

```In : a = array([[1, 2, 3, 4], [5, 6, 7, 8]])
In : a
Out:
array([[1, 2, 3, 4],
[5, 6, 7, 8]])
```

Creating an array, filled with zeroes

```In : zeros(7)
Out: array([ 0.,  0.,  0.,  0.,  0.,  0.,  0.])

In : zeros((3,3))
Out:
array([[ 0.,  0.,  0.],
[ 0.,  0.,  0.],
[ 0.,  0.,  0.]])
```

Creating an array, filled with ones

```In : ones(7)
Out: array([ 1.,  1.,  1.,  1.,  1.,  1.,  1.])
```

Creating an array, filled with sequence (similar to python's range)

```In : arange(7)
Out: array([0, 1, 2, 3, 4, 5, 6])
```

Creating an array, filled with sequence and step 3

```In : arange(2,10,3)
Out: array([2, 5, 8])
```

Creating an array with evenly spaces numbers over a specified interval

```In : start = 1
In : stop = 10
In : n = 3
In : linspace(start, stop, n)
Out: array([  1. ,   5.5,  10. ])
```

Creating an array which is filled with trash from RAM

```In : a = empty((3,3,))
In : a[:] = NAN
In : a
Out:
array([[ nan,  nan,  nan],
[ nan,  nan,  nan],
[ nan,  nan,  nan]])
```

Creating an array from vector

```In : arange(32).reshape((8, 4))
Out:
array([[ 0,  1,  2,  3],
[ 4,  5,  6,  7],
[ 8,  9, 10, 11],
[12, 13, 14, 15],
[16, 17, 18, 19],
[20, 21, 22, 23],
[24, 25, 26, 27],
[28, 29, 30, 31]])
```

Creating an array with random integer sequence

```In : size = 7
In : a = random.randint(0, 10, size)
In : a
Out: array([0, 5, 5, 7, 9, 3, 3])
```

Creating an array with random real sequence

```In : randn(7)
Out:
array([ 0.41864199, -0.97131428, -2.05212359, -0.56811645,  0.21215915,
0.17165842,  0.06305309])
```

Indexing items

```In : a = array([[1, 2, 3], [4, 5, 6], [7, 8, 9]])
In : a
Out: 8
In : a[2, 1]
Out: 8
In : a[:, 0] # first column
Out: array([1, 4, 7])
```

Indexing items with fancy indexing

```In : a[[2,8,6]]
Out: array([-1,  3,  2])

In : a = arange(32).reshape((8, 4))
In : a
Out:
array([[ 0,  1,  2,  3],
[ 4,  5,  6,  7],
[ 8,  9, 10, 11],
[12, 13, 14, 15],
[16, 17, 18, 19],
[20, 21, 22, 23],
[24, 25, 26, 27],
[28, 29, 30, 31]])
In : a[[1, 5, 7, 2], [0, 3, 1, 2]]
Out: array([ 4, 23, 29, 10])
```

Build curve, scatter and histogram

```In : a = random.randint(0, 10, 10)
In : fig = plt.figure()
In : plt.plot(range(10), a, 'k--')
In : plt.bar(range(10), a)
In : plt.scatter(range(10), a)
In : plt.savefig('figpath.svg') # save to file
```

Conditions

Output array with True values if condition is True

```In : a = random.randint(2, 15, size=10)
In : a
Out: array([12,  7,  7,  8,  5,  9,  2, 10,  3, 14])
In : a > 5
Out: array([ True,  True,  True,  True, False,  True, False,  True, False,  True], dtype=bool)
```

Combining conditions

```In : (a > 5) & (a < 10)
Out: array([False,  True,  True,  True, False,  True, False, False, False, False], dtype=bool)
```

Set value if condition is True

```In : a[(a > 5) & (a < 10)] = -1
In : a
Out: array([12, -1, -1, -1,  5, -1,  2, 10,  3, 14])
```

Creating new array where items are from a if cond is True, else items from b

```In : a = random.randint(-5, 5, size=5)
In : a
Out: array([ 0, -5,  1,  4, -3])

In : b = random.randint(-5, 5, size=5)
In : b
Out: array([-1,  2,  2, -1,  0])

In : cond = random.rand(5) > .5
In : cond
Out: array([ True,  True, False, False, False], dtype=bool)

In : where(cond, a, b)
Out: array([ 0, -5,  2, -1,  0])
```

Or we can set '+' if item is great than 0 or '-' if one less than 0

```In : where(a > 0, '+', '-')
Out:
array(['-', '-', '+', '+', '-'], dtype='|S1')
```

Sum items great than 0

```In : (a > 0).sum()
Out: 2
```

Random sequence

Creating an array with normal distribution

```a = random.normal(size=(4, 4))
```

Others functions

`permutation` - return a random permutation of a sequence, or return a permuted range

`shuffle` - randomly permute a sequence in place

`rand` - draw samples from a uniform distribution

`randint` - draw random integers from a given low-to-high range

`randn` - draw samples from a normal distribution with mean 0 and standard deviation 1 (MATLAB-like interface)

`binomial` - draw samples a binomial distribution

`normal` - draw samples from a normal (Gaussian) distribution

`beta` - draw samples from a beta distribution

`chisquare` - draw samples from a chi-square distribution

`gamma` - draw samples from a gamma distribution

`uniform` - draw samples from a uniform [0, 1) distribution

Functions

Unary functions are used for elementwise operations.

```In : a = random.randint(2, 15, size=5)
In : a
Out: array([11,  3,  6,  6, 10])
In : sqrt(a)
Out: array([ 3.31662479,  1.73205081,  2.44948974,  2.44948974,  3.16227766])
```

List of unary functions

`abs, fabs` - compute the absolute value element-wise for integer, floating point

`sqrt` - compute the square root of each element

`square` - compute the square of each element

`exp` - Compute the exponent e^x of each element

`log, log10, log2, log1p` - natural logarithm (base e), log base 10, log base 2, and log(1 + x), respectively

`sign` - compute the sign of each element: 1 (positive), 0 (zero), or -1 (negative)

`ceil` - compute the ceiling of each element, i.e. the smallest integer greater than or equal to each element

`floor` - compute the floor of each element, i.e. the largest integer less than or equal to each element

`rint` - round elements to the nearest integer, preserving the dtype

`cos, sin, tan` - trigonometric functions

List of binary functions

`add` - add corresponding elements in arrays

`subtract` - subtract elements in second array from first array

`multiply` - multiply array elements

`divide, floor_divide` - divide or floor divide

`power` - raise elements in first array to powers indicated in second array

`maximum` - element-wise maximum.

`minimum` - element-wise minimum

`mod` - element-wise modulus (remainder of division)

Statistical functions

Array initialization

```In : a = random.randn(5)
In : a
Out: array([ 1.98311057,  0.42013985,  2.77981702,  0.82146552, -0.86014544])
```

Calculate arithmetic mean

```In : a.mean()
Out: 1.0288775028049444
```

Calculate median

```In : median(a)
Out: 0.82146551999999995
```

Calculate standard deviation

```In : a.std()
Out: 1.2616131660043575
```

Calculate variance

```In : a.var()
Out: 1.5916677806355384
```

Sets

Array initialization

```In : a = random.randint(2, 10, 6)
In : a
Out: array([6, 3, 9, 4, 8, 9])
```

Intersection array

```In : in1d(a, [3,9])
Out: array([False,  True,  True, False, False,  True], dtype=bool)
```

Others functions

`unique(x)` - compute the sorted, unique elements in

`intersect1d(x, y)` - compute the sorted, common elements in x and y

`union1d(x, y)` - compute the sorted union of elements

`in1d(x, y)` - compute a boolean array indicating whether each element of x is contained in y

`setdiff1d(x, y)` - set difference, elements in x that are not in y

Save and load array to file

Save ndarray to file

```savetxt('/home/proft/temp/a.txt', a, delimiter=',')
```

```a = loadtxt('/home/proft/temp/a.txt', delimiter=',')
```

SciPy

SciPy is a collection of packages addressing a number of different standard problem domains in scientific computing. Here is a sampling of the packages included:

• `scipy.integrate` - numerical integration routines and differential equation solvers
• `scipy.linalg` - linear algebra routines and matrix decompositions extending beyond those provided in numpy.linalg
• `scipy.optimize` - function optimizers (minimizers) and root finding algorithms
• `scipy.signal` - signal processing tools
• `scipy.sparse` - sparse matrices and sparse linear system solvers
• `scipy.special` - wrapper around SPECFUN, a Fortran library implementing many common mathematical functions, such as the gamma function
• `scipy.stats` - standard continuous and discrete probability distributions (density functions, samplers, continuous distribution functions), various statistical tests, and more descriptive statistics
• `scipy.weave` - tool for using inline C++ code to accelerate array computations

Compute the determinant of a matrix

```In : from scipy import linalg
In : arr = np.array([[1, 2], [3, 4]])
In : linalg.det(arr)
Out : -2.0
```

The stats sub-package contains a wealth of functions for probability and statistics. The library currently features 81 continuous distributions and 12 discrete distributions. The following example shows how to compute the probability that a normal (Gaussian) random variable with mean 0 and standard deviation 3 takes on a value less than 5. It also shows how to generate five random samples from the same distribution.

```In : from scipy.stats import norm
In : norm.cdf(5, 0, 3)
Out : 0.9522096477271853
In : norm.rvs(0, 3, size=5)
Out : array([ 4.85229537,  3.0104119 ,  1.13189841,  5.19688369, -2.97970912])
```

We will setup ipython notebook for access from web.

Installation

```pip install numpy ipython matplotlib jinja2 pyzmq scipy
```

```# ipython

In : from IPython.lib import passwd
In : passwd()
```

Create profile and update it

```\$ ipython profile create myserver
\$ vim ~/.config/ipython/profile_myserver/ipython_notebook_config.py

c = get_config()
c.IPKernelApp.pylab = 'inline'
#c.NotebookApp.ip = '*'
c.NotebookApp.open_browser = False
c.NotebookApp.port = 9999
c.FileNotebookManager.notebook_dir = u'/home/www/ipynotebook'
```

Test run

```ipython notebook --profile=myserver
```

Configurations for nginx

```# sudo vim /etc/nginx/sites-enabled/notebook.conf

server {
server_name notebook.example.com www.notebook.example.com;

location / {
proxy_pass http://127.0.0.1:9999;
#include /etc/nginx/proxy.conf;
proxy_redirect off;
proxy_http_version 1.1;
}
}
```

```sudo service nginx reload
```

Install supervisor

```sudo apt-get install supervisor
```

Edit supervisor's config file

```# sudo vim /etc/supervisor/conf.d/sentry.conf

[program:exp]
command=/home/www/.virtualenvs/science/bin/ipython notebook --ipython-dir=/home/proft/.ipython/ --profile=myserver
directory=/home/www/ipynotebook
user=www
group=www
autostart=True
autorestart=True
redirect_stderr=True
daemon = False
debug = False
stdout_logfile=/home/www/ipynotebook/logs/supervisor_exp.log
loglevel = "info"
environment = PYTHON_EGG_CACHE="/home/www/ipynotebook/.python-eggs"
```

Update supervisor

```sudo supervisorctl reread
sudo supervisorctl update
```