Recent Posts

by John M Costa, III

My Notes On Uploading a Package PyPI

These are my notes for uploading to Pypi. Additionally, I've included some useful links that provide a lot of background.

http://diveintopython3.ep.io/packaging.html

http://wiki.python.org/moin/CheeseShopTutorial

http://packages.python.org/an_example_pypi_project/setuptools.html

  1. Register at PyPI

    You can do so here: Register at PyPI

  2. create a .pypirc file in your home directory

        vi .pypirc
    
    [distutils]
    index-servers = pypi
    
    [pypi]
    username: < username >
    password: < password >
    </pre>
    </li>
    <li><p>upload your package to PyPI</p>
    <pre>
    cd  &#060; package root &#062;
    python setup.py register sdist upload
    </pre>
    </li>
    
by John M Costa, III

New Relic's Python App Public Beta

I recently made the trek to Portland, OR for #djangocon. Demo'd there was New Relic's Real-Time Performance tool, complete with a new implementation for Python apps! This seemed like some fantastic software, but I was skeptical as to how easy it would be to install. As an experiment, I used their public beta invite on this blog.

I'd like to first point out, that the documentation to configure the the app was excellent and abundant. The software was bundled with an install file that was located in the root of their distribution (easy to find) and straight forward (easy to follow). I don't have in my notes exactly where I downloaded the installation package. However I did so and received version "newrelic-0.5.17.47.tar.gz"

Because I use Virtualenv and Pip for dependency management, I added the following line to my requirements file.

http://download.newrelic.com/python_agent/beta/newrelic-python-0.5.16.46.tar.gz

This was the most challenging part of the installation process. It wasn’t clear where this endpoint was located. While the documentation listed “http://host/path/to/newrelic-python-A.B.C.D.tar.gz” as the location, I had to ferret out the app version. Listed in the agent download page for the python agent was “http://download.newrelic.com/python_agent/beta/newrelic-python-0.5.17.47.tar.gz", however this didn’t appear to exist yet (confusing because I had already downloaded it). With a few curls, I was able to find the version listed above and carried on merrily.

Per the Install file, the next step was to create a newrelic.ini file. I copied the example file from software bundle into the root of my project. Again per the instructions, I added my settings for license_key, app_name, and log file location.

The final change I made was to my index.wsgi file. Here I added the following lines:

# configure new relic
import newrelic.agent #new
newrelic.agent.initialize('/path/to/blog/newrelic.ini')  #new

# wsgi
import django.core.handlers.wsgi # old
application = django.core.handlers.wsgi.WSGIHandler() #old
application = newrelic.agent.wsgi_application()(application) #new

The values commented as old already existed in my WSGI. Those listed where required to initialize and start the new_relic agent. A restart of the application is now required.

Now what? Start checking out the cool reports! I've shown a few examples below. Depending on the current site traffic, there may not be any data.

by John M Costa, III

My experience with python-gnupg

I was working though some usage of python-gnupg with a co-worker and, in the hope of helping out others (or my future self), am posting my shell and bpython notes here. As time permits, I'll clean up the notes.

I've broken out my notes into 4 parts:

  1. Manual Key Creation
  2. Sample File Creation
  3. Checking your keys & Writing your file
  4. Validating that it works

Manual Key Creation

I created some keys manually with gpg so that I would have a baseline to work with. If you don't have gpg installed, you can get it here

Once you have gpg installed, you can now start the process of generating your public key. Kick off the gpg generate key command. For my use, the default selections where good enough.

Johns-MacBook-Air:~ jcosta$ gpg --gen-key
gpg (GnuPG/MacGPG2) 2.0.17; Copyright (C) 2011 Free Software Foundation, Inc.
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Please select what kind of key you want\:
(1) RSA and RSA (default)
(2) DSA and Elgamal
(3) DSA (sign only)
(4) RSA (sign only)

Your selection? 1 RSA keys may be between 1024 and 4096 bits long. What keysize do you want? (2048) 2048 Requested keysize is 2048 bits Please specify how long the key should be valid.

0 = key does not expire
<n>  = key expires in n days
<n>w = key expires in n weeks
<n>m = key expires in n months
<n>y = key expires in n years

Key is valid for? (0) Key does not expire at all Is this correct? (y/N) y GnuPG needs to construct a user ID to identify your key. Real name: John Costa Email address: john.costa@gmail.com Comment: You selected this USER-ID: “John Costa john.costa@gmail.com” Change (N)ame, (C)omment, (E)mail or (O)kay/(Q)uit? O You need a Passphrase to protect your secret key. We need to generate a lot of random bytes. It is a good idea to perform some other action (type on the keyboard, move the mouse, utilize the disks) during the prime generation; this gives the random number generator a better chance to gain enough entropy. We need to generate a lot of random bytes. It is a good idea to perform some other action (type on the keyboard, move the mouse, utilize the disks) during the prime generation; this gives the random number generator a better chance to gain enough entropy. gpg: /Users/jcosta/.gnupg/trustdb.gpg: trustdb created gpg: key 6FE30238 marked as ultimately trusted public and secret key created and signed. gpg: checking the trustdb

Sample File Creation

When attempting automation, I usually try to validate that I can complete the steps manually. In this case, to validate that encryption/decryption is working, and that I haven't botched key creation, I created a sample file called "test.txt". I have placed a bit of text in the file which can be double checked when decrypted.

Johns-MacBook-Air:Documents jcosta$ echo "test" > test.txt
Johns-MacBook-Air:Documents jcosta$ cat test.txt
test

Before encrypting the file, it will be useful to know the id of the key just installed. Use the "list keys" function to display your keys.

Johns-MacBook-Air:Documents jcosta$ gpg --list-keys
/Users/jcosta/.gnupg/pubring.gpg
--------------------------------
pub   2048R/C4ECDCDC 2011-09-09
uid                  John Costa 
sub   2048R/8149FB83 2011-09-09

Now encrypt the file, outputting the encrypted file as "test.gpg". Use the public key id listed to encrypt the file.

Johns-MacBook-Air:Documents jcosta$ gpg --output test.gpg --armor --encrypt test.txt
You did not specify a user ID. (you may use "-r")

Current recipients:

Enter the user ID.  End with an empty line: John Costa 

Current recipients:
2048R/8149FB83 2011-09-09 "John Costa "

Enter the user ID.  End with an empty line:
Johns-MacBook-Air:Documents jcosta$ ls -ltr
-rw-r--r--  1 jcosta  staff    5 Sep 13 06:01 test.txt
-rw-r--r--  1 jcosta  staff  609 Sep 13 06:32 test.gpg

Now that we've encrypted a file, lets decrypt the file!

Johns-MacBook-Air:Documents jcosta$ gpg --armor --output decrypt.txt --decrypt test.gpg

You need a passphrase to unlock the secret key for
user: "John Costa "
2048-bit RSA key, ID 8149FB83, created 2011-09-09 (main key ID C4ECDCDC)

gpg: encrypted with 2048-bit RSA key, ID 8149FB83, created 2011-09-09
      "John Costa "
Johns-MacBook-Air:Documents jcosta$ ls -ltr
total 24
-rw-r--r--  1 jcosta  staff    5 Sep 13 06:01 test.txt
-rw-r--r--  1 jcosta  staff  609 Sep 13 06:32 test.gpg
-rw-r--r--  1 jcosta  staff    5 Sep 13 06:42 decrypt.txt
Johns-MacBook-Air:Documents jcosta$ cat decrypt.txt
test

====================================== Checking your keys & Writing your file

I then fired up a bpython session:

| Johns-MacBook-Air:~ jcosta$ workon example-gpg | (example-gpg)Johns-MacBook-Air:~ jcosta$ bpython

| »> import gnupg | »> gpg = gnupg.GPG(gnupghome="/Users/jcosta/.gnupg") | »> gpg.list_keys() | [{‘dummy’: u’’, ‘keyid’: u'059FF24CC4ECDCDC’, ’expires’: u’’, ’length’: u'2048’, ‘ownertrust’: u’u’, ‘algo’: u'1’, ‘fingerprint’: u'0F379C3E410B6924C2502E26059FF24CC4ECDCDC’, ‘date’: u'1315609511’, ’trust’: u’u’, ’type’: u’pub’, ‘uids’: [u’John Costa john.costa@gmail.com’]}] | »> stream = open(’/Users/jcosta/Documents/test.txt’, “rb”) | »> encrypted_ascii_data = gpg.encrypt_file(stream, “059FF24CC4ECDCDC”) | »> encrypted_ascii_data.status | ’encryption ok’

| »> encrypted_ascii_data.data | ‘—–BEGIN PGP MESSAGE—–\nVersion: GnuPG/MacGPG2 v2.0.17 (Darwin)\nComment: | GPGTools - | http://gpgtools.org\n\nhQEMAxqnnNGBSfuDAQf/est1PAn3sI4ZhPTHmcVe80wKlIcSu6N9 | BZqPykkBso9S\nfHGkcljtdJ0ICs3W38gn0qLG88UqzjNKWWCIgedAO0Pe12v38c8Ro3kN | SpJ+2hgo\nWUpn1JxuunThHyfDK8UxmNXperlO1PjKhMlFsQwSFWHhC5u7CH4/hCaVN | KOKQc0K\nkktXyoXM1D/CM1vlYCqDRbWyBdLg/W8VEOFy6zZHunDo4YxEWDmLE | EKj9kbdGTkq\ndsEL6/Y6Zykx17RMonGVCZU1X7DEyLUCuVfDGCHrlSFi8NjxFR1CB | POhJWNadzlG\nh7L8PJnWjcb/T2Mko5ZP5XWl4qN8hZljyg45x0PGzNI7AZLLnIOyzAt3T | AcyFZaJ\nhq8qxoJAvJ7tNjt4BCb1hXOav/hJ64Xyp7IpgTL1PUiC9hK7nCYwBvv3QUg= | \n=Vc4H\n—–END PGP MESSAGE—–\n’ | »> out.write(encrypted_ascii_data.data) | »> out.close()

The files aren’t exactly the same size, but they should be close.

| Johns-MacBook-Air:Documents jcosta$ ls -ltr | -rw-r–r– 1 jcosta staff 15 Sep 9 15:02 test.txt | -rw-r–r– 1 jcosta staff 592 Sep 9 16:28 test.py.gpg | -rw-r–r– 1 jcosta staff 609 Sep 9 16:33 test.gpg

Handy References

by John M Costa, III

Removing MySQL from OSX Lion

Recently I’ve had to remove a version of MySQL 5.5 from my Macbook so that I could go back to a 5.1 version. However it appears that there isn’t an automatic way to remove and install an older version. A few google searches revealed a bulk of the removal process, but additional searching revealed a few more steps.

sudo rm /usr/local/mysql
sudo rm -rf /usr/local/mysql*
sudo rm -rf /Library/StartupItems/MySQLCOM
sudo rm -rf /Library/PreferencePanes/My*
rm -rf ~/Library/PreferencePanes/My*
sudo rm -rf /Library/Receipts/mysql*
sudo rm -rf /Library/Receipts/MySQL*
sudo rm -rf /var/db/receipts/com.mysql.*

# Edit the following file, removing the line `MYSQLCOM=-YES-`.
# you may need sudo for write privileges to edit the file
# TIP: when using vim, use `dd` to delete the line and then `:wq` to save
#      the file
sudo vim /etc/hostconfig   # remove the line MYSQLCOM=-YES-

9/28/2011 - added comment on last line. Thanks Justin for pointing this out!
8/16/2013 - removed html line breaks. Added additional notes on vim from Tom Jacobs. Thanks!

Web references:

Migrating a Mercurial Repository

When I first started playing with Python and Django, I was introduced to Mercurial. I had used Subversion for a while and once familiar with Mercurial, there was no going back (well...when I had the choice ;-) ). I've posted before that I use WebFaction as a host for my personal projects. This hosting also included setting up my own Hg server. I was happy, until Ken Cochrane turned me on to BitBucket.

I've been using BitBucket off and on for about a year now. My old projects have remained in my WebFaction repository, but my new projects have been going into BitBucket. No complaints. It has been solid and reliable. I can even setup SSH public keys for all my machines accessing the account. A plus when compared with my personal hosting.

So, it occurred to me. How do I convert all my projects over to BitBucket? As with most tasks I haven't yet encountered I look to my friend Google to see if someone has solved the task in some trivial way. I should have realized how simple it was, but I'm glad I checked.

  • Create a project at BitBucket.org
  • Clone a repo from the old repository. Ensure everything is up to date with latest code and tags.
  • Change the .hg/hgrc file to point to the new Bit bucket repository

    The old .hg/hgrc file

    ``` [paths] default = https://hg.my.hosting.site.url/myproject ```

    The new .hg/hgrc file

    ``` [paths] default = ssh://hg@bitbucket.org/user account/project name ```
  • hg push

Yep, that's all it took. I love Hg.

Inspired by Andrew Frayling's post Bitbucket Import

by John M Costa, III

Django Deployment on Webfaction

Recently I deployed a django application to WebFaction (this blog!). While this wasn't the first app I've deployed there, I did forget a few steps along the way which required a bit of research and experimentation on my part. To avoid this in the future, I've documented the steps I took to deploy the app here.

I'm going to assume that you have a django app working for you locally. If you don't have one, feel free to use this sample project available for download in bitbucket.

Webfaction Control Panel

  1. After logging into the WebFaction control panel, find the Domains/websites link. Clicking on it opens up another list underneath. Click Applications. Create a new application by clicking the small icon on the bottom right of the screen.

    create_wf_app

  2. Name your application. The name is going to be the app name on the sever, so make sure that it's unix compliant. My preference is to replace spaces/dashes with underscores and lowercase everything. So "Example Blog" becomes "example_blog". The page form will validate this so that no errors are made in naming convention.

    wf_name_app

  3. Next, you'll choose the type of application you'd like to deploy. Choose the App category as Django. Next choose the App type. There are quite a few options available, some with differing versions of python. Some are listed as as insecure and shouldn't be used (unless you have a really good reason and know what you are doing). My preference is to use the latest and greatest django/python versions for new applications.

    wf_catagories_app

  4. Once you hit create, WebFaction will create the environment on their servers so that you can deploy your application. This includes python, django, and apache pre-configured to work out of the box.

    wf_catagories_app

  5. Deploying your application

  6. Do this by cloning your project into a location on the WebFaction server. WebFactions default is to place the project under the ~/webapps/. I find this long path cumbersome and usually just use my home directory. One thing I recommend is creating a symlink linking to the project. Use this symlink for any path references. This way if you want to iterate versions of code by checking out a specific tags, you need only to change the symlink.
    cd ~
    hg clone ssh://hg@bitbucket.org/jcosta71/example-webfaction-deploy example_project
    ln -s example_project example
    
  7. Apache and mod_wsgi configuration. WebFaction creates a default apache2 configuration file and project wsgi file. For the sake of this example, I've pulled them into the example project. All that should be required is to correct the pathing in these two files, create symlinks to the appropriate locations, and restart apache.
    cd ~/webapps/example_application
    ln -s ~/example/apache/conf/example.wsgi example.wsgi
    cd ~/webapps/example_application/apache2/conf
    mv httpd.conf httpd.conf.bak
    ln -s ~/example/apache/conf/httpd.conf httpd.conf
    

    Because of my preference to checkout the applications into the root. I’ve tweeked the wsgi and httpd.conf files slightly. The changes include:

    example.wsgi:

    From:

    os.environ['DJANGO_SETTINGS_MODULE'] = 'myproject.settings'
    

    To:

    os.environ['DJANGO_SETTINGS_MODULE'] = 'settings'
    

    httpd.conf:

    From:

    WSGIPythonPath /home/johncosta/webapps/example_application:/home/johncosta/webapps/example_application/lib/python2.6
    WSGIScriptAlias / /home/johncosta/webapps/example_application/myproject.wsgi
    

    To:

    WGIPythonPath /home/johncosta/example/sample_project/:/home/johncosta/webapps/example_application/lib/python2.6
    WSGIScriptAlias / /home/johncosta/webapps/example_application/example.wsgi
    
  8. Add the static files

  9. The next step is to generate and map the static files to your application. Django has added in 1.3 static directory support. See the documentation here. To map your static files to the application you'll first need to publish them in a known location.

    1. Set the STATIC_ROOT value in your local_settings.py file to the location you want all the static files collected to.
    2. Change your directory to the location of your manage.py file.
    3. Run the collect static management command. You may have to add django to your python path.
    cd ~/example_project/sample_project</li>
    PYTHONPATH=/home/johncosta/example/sample_project/:/home/johncosta/webapps/example_application/lib/python2.6/:$PYTHONPATH
    python2.6 manage.py collectstatic
    

  10. Create a new application for your static content by going back into the WebFaction Control Panel. Name your new application and select symbolic link for a category. You'll use the default symbolic link to static/cgi/php app type. In the extra info, enter the absolute path to the path you set your STATIC_ROOT.

    /home/johncosta/example/sample_project/static_root
    

    wf_create_static

  11. It's alive!

  12. This last step maps your application and static content to a domain. Log back into WebFaction control panel, choose Websites under Domains/websites. I've already created a domain in the control panel that I'd like to map my application to. Webfaction has a how-to guide on creating domains here.

    I've chosen to name the example_projects as example. Choose the subdomain for your application, and map your site app (we call this example_application) to "/". Map the static app (this was named example_application_static) to "/static". This value should be the same value as STATIC_URL found in your local_settings.py file.

    wf_map_static

  13. You should now be able to access your domain! Here's my example. http://example.johncosta.webfactional.com
by John M Costa, III

django-sitemap-module-object-has-no-attribute-valu

I ran into an issue trying to generate a sitemap.xml file with Django’s built-in sitemap view. After reading the documentation over a few times I still received the error:

Django sitemap: 'module' object has no attribute 'values'
For some reason, it wasn't obvious to me what the confusion was, but clearly others have encountered the issue as well. A quick google search turned up a question on StackOverflow. User KuB had asked a question which looked very similar to the one I had just experienced.

After a bit of research and experimentation I was able to resolve the issue and came up with a bit of code that resolved the issue for me:

The full coding sample that I used looks like the following:

sitemap.py file:

from django.contrib.sitemaps import Sitemap
from articles.models import Article

class BlogSitemap(Sitemap):
    changefreq = "never"
    priority = 0.5

    def items(self):
        return Article.objects.filter(is_active=True)

    def lastmod(self, obj):
        return obj.publish_date

urls.py file:

from sitemap import BlogSitemap

# a dictionary of sitemaps
sitemaps = {
    'blog': BlogSitemap,
}

urlpatterns += patterns (''
    #...<snip out other url patterns>...
    (r'^sitemap\.xml$', 'django.contrib.sitemaps.views.sitemap', {'sitemaps': sitemaps}),
)

You can find the full thread out on Stack Overflow here.

by John M Costa, III
by John M Costa, III