DNS Configuration

This post answers the question "how do I host my email at one provider and my web site at another?". In particular it shows how to set up DNS at your domain name registrar, email at Fastmail and web hosting at WebFaction.

Most of the how to DNS configuration guides on the web show how to point to a single hosting provider for web, email and other services. I'm awkward and wanted to separate my email from web hosting. So by a little trial and error I came up with this set up for my personal domain.

First at your domain registrar (I use hover) set the domain namervers to Fastmail, I've used 2 of their name servers.

Nameservers

This delegates control of all DNS activity for the domain to Fastmail. Without any other changes this will allow Fastmail to manage email and serve a limited web site from their hosting option. I choose to host my web site at WebFaction and so to delegate this from Fastmail we use their domain configuration dashboard and set up DNS 'A' records to point to my WebFaction host.

A records

The final step is to configure WebFaction to accept and serve your domain. If they don't recognise incoming web traffic you won't be able to server a web site on your domain. Here's mine set up in domains section of the WebFaction control panel.

Control panel

RSS Readers in 2016

After the great Google Reader shutdown of 2013 I switched my feed reading to a self-hosted instance of Fever and I've been a happy user ever since. I didn't mind that it wasn't open source, I was glad to spend the money to support a developer building a great product. Sadly now that support and developed has ended. I'd like to thank Shaun Inman for a great product. The end of development and support for Fever don't necessarily mean that it will stop working at any time in the near future. But I took this as an opportunity to look around at the state of the RSS reader world and see if there isn't a better tool for me to use now and for the future.

My requirements are simple. It must run on a server that I control, be accessible using a reasonably modern web browser and support import and export of feed listings via OPML. Theere are bonus marks if the Fever API is supported (I've got Reeder running on my iPad) and for ease of installation and use. I'm not a sophisticated feed reader but I've accumulated quite a collection of sites that I follow over the years. Fever is currently following 370 feeds but I can use any transition as an opportunity to trim away the dead wood.

After a little research and with some help from OpenSource.com I narrowed my search to three possible solutions: selfoss, miniflux and Tiny Tiny RSS

My evaluation strategy was pretty simple - install the tool on an Ubuntu virtual machine and if it looks promising do a fresh install on my web server and try the tool for size for a few days.

Selfoss

This topped a few of the searches I did and was the first solution I installed. Well, I say that but I haven't managed to install it on my VM. I got to the stage of posting to the forum asking about error messages that reported missing PHP .so files before I gave up. In this particular case it looks to me like the install expected PHP 5 but on the Ubuntu 16.04 the default is PHP 7. I may be wrong, I'm certainly no PHP guru, but clear installation requirements and instructions are nice. I may revisit selfoss again and see if I can't get over my installation problems.

Miniflux

Miniflux takes the river of news approach and applies it to a minimalist presentation. Installation was fairly straight forward and I managed to import my feed collection quite quickly. The folder structure that I had set up in Fever didn't make the transition - it isn't part of the OPML contents - so I took the opportunity to trim my feed list and put them into a small number of groups.

Updating the feeds is simple in the web UI - just press the 'refresh all' link on the subscriptions page. Then go to the 'unread' page and scroll through the articles. There are some nice simple keyboard shortcuts to aid navigation and reading.

There are 2 puzzles I still haven't cracked with Miniflux. I can't get the automated feed refresh working - probably something to do with the way I have set up PHP - and I still haven't found out what the 'groups' are used for. Apart from this minor quirks I am liking the philosophy and execution of miniflux and I think it's going to be my feed reader from now on.

Tiny Tiny RSS

Tiny Tiny RSS has been around for a few years and receives rave reviews on the web. Installation on my virtual machine was pretty straight forward and I had it up and running without any problems. The interface is more complex than Miniflux, and even than Fever, with nested folders of feeds and a nice display of each article.

Where I struggled with Tiny Tiny RSS was refreshing the feeds. The documentation says that this is best done via a cron job but I couldn't get the command working. I'm also not mad keen on running a PostgreSQL server on my web host as I'm not knowledgeable enough to keep it running securely.

So for the time being I'm going with Miniflux

Simple Log

I found myself putting the same few lines of code at the top of every script or module that I wrote. It went something like this:

import logging
MESSAGE_FORMAT = "%(asctime)s %(levelname)s:: %(message)s"
DATE_FORMAT="%Y.%m.%d %T"
logger = logging.getLogger(self.__name__)
logger.setLevel(logging.INFO)
formatter = logging.Formatter(MESSAGE_FORMAT, DATE_FORMAT)
ch = logging.StreamHandler()
ch.setFormatter(formatter)
logger.addHandler(ch)

Because the default simple logging provided by the standard library logging package doesn't do everything I'd like. I prefer that messages contain the time that they were generated. I prefer that the default logging level is set to INFO. I don't really care to see the log name in each log message.

When I built bigger applications I found myself putting this code in a log.py file and including it with the application source code. Then I wanted to change the log format. Which meant editing quite a few copies of the same file. So I decided that was a bad idea and I would turn my simple few lines of code into a Python module and publish it on PyPi. That way I could include a single copy of the code in each of my modules or applications with a simple pip install command.

The last time I did some serious packaging work this all rather tricky but thanks to the most excellent work of the Python Packaging Authority publishing your own Python module is a breeze these days.

So say hello to simple_log. It's available from PyPI now - https://pypi.python.org/pypi/simple-log/

The documentation is on this site - http://halfcooked.com/code/simple_log/ and the source code lives on BitBucket at https://bitbucket.org/andy47/simplelog

It's designed to be a simple log module that can be incorporated in any Python application and used as simply as:

>>> from simple_log import get_log
>>> log = get_log()
>>> log.info('Information message')
2016.12.12 13:30:30 INFO:: Information message

No more worrying about streams and formatters just a nice simple way to get a log object and log messages from your code. The code is licensed under the MIT license so corrections, additions and praise are always welcome.

Generating Diceware Passwords in Python

Today I'm going back to a theme from a post I wrote last year and looking at generating passwords with my favourite programming language. A tweet from Simon Brunning pointed me to Micah Lee's article at The Intercept and my first thought was to write a function to do this in Python. So here it is;

import random
word_dict = {}
passphrase = []
with open('diceware.wordlist.andy.txt') as f:
    for line in f.readlines():
        index, word = line.strip().split('\t')
        word_dict[int(index)] = word

for words in range(0, word_count):
    this_index = 0
    for position in range(0, 5):
        digit = random.randint(1, 6)
        this_index += digit * pow(10, position)
    passphrase.append(word_dict[this_index])
return ' '.join(passphrase)

In terribly bad form I've hard coded the diceware word list file name. I took the English word list and converted it to a plain text file for easier processing. The original will probably work just as well, I just haven't tested it.

Teaching an old dog to Fish

Inspired by the recent furore around Shellshock I decided that it was time to try an alternative to bash By the very grown up process of shutting my eyes and poking my finger at the results of a search for "shell" I ended up at fish shell, described by the project as "a command line shell for the 90s". I'm presuming that this means the 1990s and is not a minimum age requirement.

I'm quite enjoying it so far but the missing piece for me was something equivalent to the very useful virtualenvwrapper. Luckily for me someone else has already had this problem and wrote virtual fish. For which many thanks. I did have a little trouble with installation and configuration. I'm sure that this was entirely my own fault but as a reminder for me and anyone else who stumbles on this here is what I did.

  • Downloaded virtual.fish from the Github repo to ~/bin
  • Because I don't use ~/.virtualenvs to store my virtual environments I added set -x VIRTUALFISH_HOME ~/Work/envs/ to my .config/fish/fish.config file
  • Included the currently active virtual environment name in my prompt by following the instructions. A trap for young players is to make sure that you do funcsave fish_prompt as instructed. I missed this out the first couple of times and kept wondering why my prompt was wrong.

Update: Thanks for the comments. bronsen - yes I do source virtual.fish in my config.fish. Or at least I did until jl pointed me at Pew. Now I've switched to that from virtualenvwrapper and virtual.fish and I must say that it works like a dream. Thanks both for taking the time to leave a comment.

Generating Reasonable Passwords with Python

Thanks to a certain recent Open SSL bug there's been a lot of attention paid to passwords in the media. I've been using KeePassX to manage my passwords for the last few years so it's easy for me to find accounts that I should update. It's also a good opportunity to use passwords that are stronger than words such as 'banana'.

My problem is that I have always resisted the generation function in KeePassX because the resulting strings are very hard to remember and transcribe. This isn't an issue if you always use one machine but I tend to chop and change and don't always have my password database on the machine I'm using. I usually have a copy on my phone but successfully typing 'Gh46^f27EEGR1p{' is a hit and miss affair for me. So I prefer passwords that are long but easy to remember, not unlike the advice from XKCD.

Which leaves a problem. Given that I now have to change quite a lot of passwords how can I create suitably random passwords that aren't too difficult to remember or transcribe? Quite coincidentally I read an article titled Using Vim as a passowrd manager. The advice within it is quite sound and at the bottom there is a Python function to generate a password from word lists (in this case the system dictionary). This does a nice job with the caveat that as I understand it from a cryptographic standpoint the passwords it creates are not that strong. But useful enough for sites which aren't my bank or primary email. For those I'm using stupidly long values generated from KeePassX. When I tried the Python function on my machine there was one drawback, it doesn't work in Python 3. This is because the use of 'map' is discouraged in Python 3. But that's alright because I can replace it with one of my favourite Python constructs - the list comprehension. Here is an updated version of invert's function that works in Python 3. Use at your own risk.

def get_password():
    import random
    # Make a list of all of the words in our system dictionary
    f = open('/usr/share/dict/words')
    words = [x.strip() for x in f.readlines()]
    # Pick 2 random words from the list
    password = '-'.join(random.choice(words) for i in range(2)).capitalize()
    # Remove any apostrophes
    password = password.replace("'", "")
    # Add a random number to the end of our password
    password += str(random.randint(1, 9999))
    return password

It'll Come Out in the Wash

After reading this fine summary of the history of Python 3 by Nick Coghlan I was inspired to update as many of my half finished projects and miscellaneous scripts as possible. Then I looked up and I had lost several hours of my life. To save random internet strangers from the same pain as I experienced here is a catalogue of problems and how I solved them.

tl;dr - Python 3.3 on Ubuntu 12.04 LTS is possible but quite tricky. Unless you like system administration you are probably better off waiting until 14.04 LTS comes out in a few months.

I've settled on a default 'stack' of Python and related technologies that I use for most of the code that I write these days;

The good news is that all of the Python related components of these technologies have been ported to Python 3. The bad news is that Flask only runs on Python 3.3. Elementary OS is based on Ubuntu 12.04 LTS and that only has Python 3.2 in it's repositories.

So, first things first, how do we get Python 3.3 on Elementary OS? Or Ubuntu 12.04 which are interchangeable for the purposes of the rest of this post. Quite easily thanks to the 'deadsnakes PPA' as explained in this askubuntu post.

I also don't like to pollute my system Python or operating system packages with project specific Python modules so I always use virtualenv. Unfortunately this doesn't work with the Python 3.3 acquired from the deadsnakes PPA as that doesn't ship with easy_install.

So by this stage I have Python 3.3 running but no way to create or manage a virtual environment. I then remembered Nick's talk at PyCon AU where he mentioned that a new lightweight virtual environment would be included in the standard library. Sure enough the venv module (and pyvenv script) were present on my system so I decided it was a good idea to use them.

The bad part for me is that (and it says this in the documentation) venv does not include any means of installing other Python packages. How then do you get a package installer/manager when you can't install packages? The solution was in a c.l.p. post.

That link describes a process of creating a new virtual environment and then bootstrapping distribute. In case my link-fu fails me the simple shell based approach is;

$ pyvenv py3k ~/envs/py3k
$ cd py3k
$ source bin/activate
$ wget http://python-distribute.org/distribute_setup.py
$ python distribute_setup.py 
$ ./local/bin/easy_install pip

One final little wrinkle though. The c.l.p. post and other references I found indicate that your package scripts (such as 'pip' and 'python') will be installed in $ENV_HOME/bin. In my environment they are installed in $ENV_HOME/local/bin. It's probably something I have inadvertently done. But if you follow the instructions and can't find something in bin have a look in local/bin and you should find it.

Playing with Software

I've been trying to install open source software. In this case an application called Moodle. I followed the instructions to install it on an Ubuntu Server virtual machine but they were missing a couple of key points.

After installing the Moodle package (under "Moodle Installation" in the instructions) you need to copy the generated Apache configuration file into your Apache conf.d directory. To do this try the following command;

$ sudo ln -s /etc/moodle/apache.conf /etc/apache2/conf.d/moodle.conf

As you can see from this handy guide to configuring Apache on Debian application specific configuration files should go in the conf.d directory rather than hacking the httpd.conf file.

The second problem is that the default http://locahost/moodle URL won't work on a server because access is restricted to the local machine only by default. To allow access from other machines (in my case the host computer) you need to edit the generated apache.conf file and uncomment the line which says allow from all. This will enable remote access to the instance from your host machine.

How To Start Unit Testing

I had the great privilege to present this weekend at PyCon Australia 2013. My talk was originally titled "Why I Use py.test and Maybe You Should Too" but as I wrote the paper and accompanying slides I realised that it should really have been called "How to Start Unit Testing" as that was really the key message from my talk.

I've uploaded the slides and the paper I wrote to accompany them to the presentations section of this site. Feedback and corrections are alway welcome. Thanks to the PyCon Australia committee, volunteers and delegates for another great conference.