Andrew Channels Dexter Pinion

Wherein I write some stuff that you may like to read. Or not, its up to you really.

April 15, 2005

Bad Kinterbasdb

In my search for the perfect desktop RDBMS I installed Firebird and it's DB-API companion KinterbasDB.

I tested them under the most extreme of circumstances by seeing if I could get them installed and running on my laptop during the thirty minute ferry ride to work.

Following these instructions I had Firebird installed and a database created in no time. I then installed KinterbasDB quite easily and fired up my python interpreter.

>>> import kinterbasdb
>>> myDb = kinterbasdb(user='sysdba', password='masterkey', database='~data/firebird/firstdb.fdb')
ImportError: No module named mx

Hmm, that's a little surprising. On consulting the documentation it looks like you really need to use the mx.DateTime third party module. The alternative is quite hairy scary looking and frankly puts me off using the module. If I had Firebird databases I wished to connect to from Python then I'd probably persevere. But my thirty minutes were nearly up and I hadn't managed to connect to my database yet.

When I got to work, and in the interests of fair play, I did a bit more reading. According to this gem I found in the documentation you can rely on the standard datetime module, but to do that you need another third party module (fixedpoint). I gave up at that point as my time was well and truly up.

So my suggestion to the kinterbasdb team would be to provide a type_conv option which only relies on modules in the standard library, this way the casual dabbler who tries out your module will be able to use it.

Posted by Andy Todd at April 15, 2005 08:34 AM

Comments

I agree entirely. I believe that longer-term, the intention is to move to stdlib classes as the default (I can't recall where I saw this, it may have been on the mailing list). Obviously, there's a backward compatibility problem here...

It put me off KinterbasDB, as well. (Maybe I'll give it another go, though, when I feel a bit more adventurous).

Posted by: Paul Moore on April 15, 2005 06:26 PM

A simple question: Don't you look into the documentation before you try out?
I am using the kinterbasdb-module since it has been version 2.something and Interbase 5.5, and IMHO it is a rock-solid team in the worlds of windows and *nix.


Posted by: Marcus Bajohr on April 16, 2005 12:40 AM

Marcus, I did what ninety five percent of users do; install the software and then look at the manual when you find a problem.

Regardless, it's not possible to talk to an Interbase (or Firebird) database without at least one third party module. That was a deal breaker for me.

Add to that the fact that storing the entire database in a single file isn't big or clever.

Posted by: Andy Todd on April 16, 2005 12:12 PM

My opinion: downloading and installing the mx module takes less then 1 minute.
Have you tried it before you'v written "Bad kinterbasdb" ?

But. Please send your suggestion to David Rushby
http://sourceforge.net/forum/forum.php?forum_id=30917

He is wery helpful!

Posted by: Bajusz Tamás on April 16, 2005 09:14 PM

Actually, I'm aware of and understand the original poster's usability complaints; I certainly considered them when making the design decisions that gave rise to them.

The problem is that kinterbasdb has existed for many years, and supports old Python versions, but the Python standard library didn't provide convenient representations for all of the required types (namely, date/time/timestamp (the datetime module) and numeric/decimal (the decimal module)) until Python 2.4.

So although I wholeheartedly agree that it should be easy to load a set of type translators that relies solely on the standard library (and the CVS version makes it as easy as 'kinterbasdb.init(type_conv=200)'), a large amount of existing code would break if this were made the default behavior.

I know very well that first impressions are crucial to the adoption of software, but frankly, backward-compatibility for the users who have huge investments in kinterbasdb is more important than friendliness to users who have next to no investment. It's too bad we can't have it both ways.

Posted by: David Rushby on April 18, 2005 04:42 PM

David, thanks for replying. I can think of a couple of things;

1) Try gracefully importing;

try:
    import datetime
except ImportError:
    import mx

Although I admit this isn't exactly backwardly compatible so maybe switch it around?

try:
    import mx
except ImportError:
    import datetime

2) I didn't ask that a new type_conv option be the default, just that it might be nice to have one available if my first suggestion is a no no. Including a type_conv value of 200 would just be an option for investment-lite users like myself. As far as I can tell it wouldn't break any existing code.

Posted by: Andy Todd on April 18, 2005 06:39 PM

#1 isn't backward-compatible enough. Suppose you're a long-time kinterbasdb user creating a new installation of an application written a couple of years ago that uses mx, but you forget to install mx. Now, instead of an ImportError near the time the app starts, you get a potentially confusing error later.

Alternatively, suppose you're a new kinterbasdb user with a production server running the system Python 2.4 on a Linux distribution that has mx installed right out of the box. Suppose you're developing on a Windows workstation, but since you needed to install packages manually there and have no explicit desire to use mx, you didn't install mx. If kinterbasdb implemented #1, the development workstation would default to stdlib datetime, but the production server to mx--bad news.

#2 is indeed a good idea, wouldn't break any existing code, and has been available in CVS HEAD since December or thereabouts. Python 2.4 arrived too late in the kinterbasdb 3.1 development cycle for new Py2.4-oriented features to be included in it, and I haven't officially initiated the 3.2 development cycle yet.

Probably the best compromise between the concerns of new and existing users is to make references to the type_conv convenience codes in general, and code 200 in particular, much more prominent in the documentation (there's currently no mention of option 200 because it doesn't exist in kinterbasdb 3.1).

Anyway, thanks for your constructive criticism.

Posted by: David Rushby on April 18, 2005 08:43 PM

>> Add to that the fact that storing the entire database in a single file isn't big or clever. <<

Actually this comes in really handy for copying IB/FB databaes around. Disconnect all users from the DB and just copy it. Much faster than doing a full backup-restore cycle which some databases force you to do.

But in any case Firebird has so much to offer that it doesn't make sense to dismiss it for this reason. Look closer, you'll be glad you did.

Posted by: Ian Sparks on April 19, 2005 11:18 PM

No Ian, it's a really bad idea.

1. It puts a practical limit of 2Gb on the database size as that's the maximum file size on most operating systems, certainly the ones that I use.

2. Having all of your data in a single file increases the likelihood of contention when there is more than one process accessing the database. Having multiple data files doesn't eliminate this but does allow your data to be more widely spread over your file system which reduces the probability of it happening.

What in particular, and I am interested, does Firebird offer over say MySQL or PostgreSQL?

Posted by: Andy Todd on April 20, 2005 09:08 AM

If you look closer at FB, it does support splitting the db file into secondary files. Check the CREATE DATABASE statement. Another interesting feature is Shadow, which gives you database mirrors--check CREATE SHADOW statement.

What I really like about FB?
1. It's small footprint
2. Fast
3. Very low administrative overhead

Posted by: Arthur Lee on April 20, 2005 11:17 AM

I have the mx module installed, but don't use it with kinterbasdb - I use it with the datetime objects.

I have been quite pleased with kinterbasdb - I just followed the documentation to get the datetime stuff going. It looks scarier than it is.

I'll second what Arthur Lee says about Firebird. We use it for a local database in our application. It is small, lightweight, embeddable, etc. Stored procedures, triggers, database events, etc. are all there if you need them. My experience with InterBase goes back to 1993...

MySQL - Until recently, my understanding is that it did not have triggers, referential integrity, etc. There is also the license to deal with for commercial apps.

Posted by: Hubert Hickman on April 20, 2005 12:04 PM

Responding to David Rushby's comments, would a 3.1 patch release, incorporating stdlib support, be an option?

I run on Windows, and building binaries is a pain, so CVS isn't a practical option for me.

I use dates a lot, and for "normal" processing, rely on the stdlib datetime module. So using mx.Datetime for database return values would entail a lot of unnecessary type conversion, for no benefit. The datetime support is key for me. Support for decimal is far less important, but "stdlib-only" behaviour is important, as I work in environments where I cannot always install additional modules.

As none of the integer arguments to kinterbasdb.init do what I want, I need (currently) to go with the "object variant", but I found it pretty much impossible to determine from the documentation how to do this - specifically, how I might create a parameter which does datetime like 100, but decimals like 'FLOATING' or 'FIXED' (not sure which I'd prefer - it probably depends on the application). Can anyone enlighten me? (And a simple example like this could probably do with being added to the documentation).

Posted by: Paul Moore on April 20, 2005 09:12 PM

Actually, this looks like it works:

--- put this in kb_conv.py (with appropriate indentation...) ---
from kinterbasdb import typeconv_datetime_stdlib
from kinterbasdb import typeconv_fixed_stdlib
from kinterbasdb import typeconv_text_unicode

_underlying_modules = (
typeconv_fixed_stdlib,
typeconv_datetime_stdlib,
typeconv_text_unicode # 2004.01.03
)

# Load the required members from the underlying modules into the namespace of
# this module.
globalz = globals()
for m in _underlying_modules:
for req_member in m.__all__:
globalz[req_member] = getattr(m, req_member)
del globalz
---

Now do

import kinterbasdb
import kb_conv
kinterbasdb.init(kb_conv)

... and you should have Python 2.3 datetime support, and something using only stdlib (I'm not 100% clear what...) for numbers.

Fairly black magic, but it seems to work...

Posted by: Paul Moore on April 20, 2005 11:06 PM

>
>No Ian, it's a really bad idea.

1. File size limit.
2. Contention issues
<<

As Arthur pointed out, FB databases can be split into as many files as you want. This breaks my "copy the database around" usefulness point but at least meets your issue.

Regarding contention issues in a single file. Well maybe. I won't pretend I'm smart enough to rebut that. Ask on the firebird list. All I can say is that I've been using IB since 1998 in a commercial setting and it has been great for us primarily because of it's set-and-forget zero-maintenance and more than adequate performance.

I won't rehash the IB vs MySQL vs PostgreSQL debate. It all depends on what you want to do. Clearly you have some use for a DB in mind and you're not super-happy with your existing database options or you wouldn't be giving FB a try.

My guess is you'd want to use FB where you need more features than MySQL and want to do less maintenance than you'd have to do with PostgreSQL (i.e. none) or where you really need to be able to trust the performance/stability of the database on both Windows and Linux (not knocking PostgreSQL 8 but it only just made it to Windows and the Cygwin stuff frankly doesn't count).


Posted by: Ian Sparks on April 21, 2005 05:23 AM