IOPro

IOPro loads NumPy arrays (and Pandas DataFrames) directly from files, SQL databases, and NoSQL stores, without creating millions of temporary, intermediate Python objects, or requiring expensive array resizing operations. It provides a drop-in replacement for the NumPy functions loadtxt() and genfromtxt(), but drastically improves performance and reduces the memory overhead.

IOPro is included with Anaconda Workgroup and Anaconda Enterprise subscriptions.

To start a 30-day free trial just download and install the IOPro package.

If you already have Anaconda (free Python distribution) installed:

conda update conda
conda install iopro

If you do not have Anaconda installed, you can download it here.

IOPro can also be installed into your own (non-Anaconda) Python environment. For more information about IOPro please contact sales@continuum.io.

Getting Started

Start by attaching to a data source (in this case, a local csv file):

>>> import iopro
>>> adapter = iopro.text_adapter('table.csv', parser='csv')

We can specify the data types for values in the columns of the csv file being read though here we will instead rely upon the ability of IOPro’s TextAdapter to auto-discover the data types used.

We ask IOPro’s TextAdapter to parse text and return records in NumPy arrays from selected portions of the csv file using slicing notation:

>>> # read first ten records
>>> array = adapter[0:10]

>>> # read last five records
>>> array = adapter[-5:]

>>> # read every other record
>>> array = adapter[::2]

>>> # read first and second fields only
>>> array = adapter[[0,1]][:]

>>> # read fields named 'f2' and 'f4' only
>>> array = adapter[['f2','f4']][:]

Requirements

  • python 2.6, 2.7, or 3.3+
  • numpy (>= 1.6)

Python modules (optional):

  • boto (>= 2.8)
  • numba (>= 0.8)

Release Notes