Setting up python development environment with buildout

Attn: Checkout Conda before trying this.

Buildout is a Python-based build system for creating, assembling and deploying applications from multiple parts, some of which may be non-Python-based. It lets you create a buildout configuration and reproduce the same software later.

I’ve documented the steps required to create a simple buildout based project.

  1. Start by creating a project directory and initialise a virtual environment inside the project directory.

    $ mkdir word_count_buildout && cd word_count_buildout
    $ virtualenv --no-site-packages .env
    $ source .env/bin/activate
  2. Fetch A common script required to create necessary directories and eggs (like setuptools, etc. )

    $ wget
  3. Create a buildout configuration file

    $ vi buildout.cfg

    copy & paste the following snippet

    develop = .
    parts = job job-test scripts zipeggs
    # index =
    recipe = zc.recipe.egg
    interpreter = python
    eggs = wordcount-job
    recipe = pbp.recipe.noserunner
    eggs = pbp.recipe.noserunner
    working-directory = ${buildout:directory}
    recipe = zc.recipe.egg:scripts
    eggs = dumbo
    recipe = zipeggs:zipeggs
    target = dist 
    source = eggs

    basic config file structure explained here.

    • ln:4 index If using private PyPi, uncomment and replace the URL.
    • ln:21 recipe = zipeggs:zipeggs – a buildout recipe to zip all flattened/unzipped eggs, flattened/unzipped eggs are convenient while developing (for debugging purpose) and they load faster. Dumbo requires zipped eggs to be passed via -libegg param, zipeggs recipe can generate zipped eggs under target directory (dist). more details @tamizhgeek repo.
  4. Execute the bootstrap file

    $ python
    Extracting in /tmp/tmpZ7ki33
    Now working in /tmp/tmpZ7ki33/setuptools-18.1
    Building a Setuptools egg in /tmp/bootstrap-q4Xfc5
    Creating directory '/home/raj/workspace/word_count/eggs'.
    Creating directory '/home/raj/workspace/word_count/bin'.
    Creating directory '/home/raj/workspace/word_count/parts'.
    Creating directory '/home/raj/workspace/word_count/develop-eggs'.
    Generated script '/home/raj/workspace/word_count/bin/buildout'.

    The script has created few directories and a buildout script inside bin directory. Read more about the directory structure here

  5. Create

    $ vi
    from setuptools import setup, find_packages
    import os
    version = os.environ.get("PIPELINE_LABEL", "1.0")

    Read more about setuptools here.
    Any changes to and buildout.cfg should be followed by executing ./bin/buildout

  6. Create a python module for a simple word-count dumbo job.

    $ mkdir wordcount-job && touch wordcount-job/
    $ vi wordcount-job/

    copy & paste the following code.

    def mapper(key, value):
        for word in value.split():
            yield word, 1
    def reducer(key, values):
        yield key, sum(values)
    if __name__ == "__main__":
        import dumbo, reducer)
  7. Finally, run the buildout script to fetch artifacts from private or central pypi

    $ ./bin/buildout

    All the dependencies (and its dependencies) mentioned in are collected under eggs/ directory. Zipped eggs are available under dist/ directory and executable scripts with dependencies wired are generated under bin directory. Try viewing the contents of bin/dumbo.

  8. To run the job

    $ ./bin/dumbo start wordcount-job/ -input /tmp/input -output /tmp/output 
 can access all the dependencies (under eggs/ directory) mentioned in

  9. To run tests.

    $ ./bin/job-test

    read more about pbp.recipe.noserunner recipe and nose

  10. To build egg

    $ ./bin/buildout setup . bdist_egg
  11. Private PyPi repos can also be used to distribute python eggs. To publish an egg to a private pypi, create config for pypicloud under home directory

    $ cd $HOME && vi .pypirc

    copy & paste the following

    index-servers = my-pypi
    username: username
    password: password

    Under project working directory

    $ ./bin/buildout setup . bdist_egg upload -r my-pypi
  12. Source code
  13. Beer to bud @azhaguselvan for support and stuff.