Txredisapi: Adding Support for WATCH/UNWATCH Commands

txredisapi [1] is a “redis client for twisted”. I believe it supports everything redis has to offer. The only drawback I found is that transaction support is somewhat limited, as there is no explicit WATCH command.

1
2
3
4
>>> conn = ...
>>> tx = yield conn.multi("foobar")
>>> yield tx.set("foobar", 20)
>>> yield tx.commit()

In this example, the multi function invokes WATCH and gives you no chance to perform any operation prior issuing MULTI. To see the problem, suppose implementing the INCR command. The current API probably makes this an impossible task.

Starting today, this is no longer! I’ve submitted a patch to fix it and if it gets accepted:

1
2
3
4
5
6
7
8
>>> conn = ...
>>> tx  = yield conn.watch("foobar")
>>> old = yield tx.get("foobar")
>>> if (old is None):
>>>     old = 0
>>> yield tx.multi()
>>> yield tx.set("foobar", old + 1)
>>> yield tx.commit()

Yay, full redis transactions! And the implementation was remarkable simple. For more information, refer to the following:

[1]https://github.com/fiorix/txredisapi

Python Twisted and Deferreds

Today I faced a rather awkward feature of twisted deferreds. Consider the folliwng code:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
>>> import sys
>>> import twisted
>>> from twisted.internet import defer
>>>
>>> twisted.version
Version('twisted', 12, 2, 0)
>>>
>>> o = lambda s: sys.stderr.write("%s\n" % s)
>>> d = defer.Deferred()
>>>
>>> d.addCallback(lambda _: o("on callback"))
>>> d.addErrback(lambda _: o("on errback"))
>>>
>>> d.errback(RuntimeError())
on errback

Unsurprisingly, the error callback got invoked. The following was completely unexpected, tough:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
>>> import sys
>>> from twisted.internet import defer
>>>
>>> twisted.version
Version('twisted', 12, 2, 0)
>>>
>>> o = lambda s: sys.stderr.write("%s\n" % s)
>>> d = defer.Deferred()
>>>
>>> d.addErrback(lambda _: o("on errback"))    # mind the
>>> d.addCallback(lambda _: o("on callback"))  # order
>>>
>>> d.errback(RuntimeError())
on errback
on callback

Both callbacks got invoked this time, even though the second one was not an error callback.

Twisted code uses the type of the parameter to figure out which callback type to call:

The addErrback and addCallback are just helpers that wrap around addCallbacks using some identity function [passtru]. Using both functions, instead of one you end up with two functions for each callback type:

Then, depending on the order you will either have:

# addCallback; addErrback
passtru -> lambda

# addErrback; addCallback
lambda -> passtru

The callbacks are all chained, the return value of the current callback is the the argument of the next one. In this case, that lambda expression produces a None value, and the next callback to be invoked is no longer the error callback.

I̶ c̶o̶u̶l̶d̶n̶’̶t̶ f̶i̶n̶d̶ a̶n̶y̶t̶h̶i̶n̶g̶ m̶e̶n̶t̶i̶o̶n̶i̶n̶g̶ t̶h̶a̶t̶ b̶e̶h̶a̶v̶i̶o̶r̶ o̶n̶ t̶h̶e̶ d̶o̶c̶u̶m̶e̶n̶t̶a̶t̶i̶o̶n̶.̶ M̶a̶y̶b̶e̶ t̶h̶i̶s̶ i̶s̶ i̶n̶d̶e̶e̶d̶ a̶ b̶u̶g̶.̶

UPDATE: It is a very well documented feature actually. I somehow missed that:

In particular:

[…] If an errback doesn’t raise an exception or return a twisted.python.failure.Failure instance, switch to callback.

Dot Files Reloaded

It’s been a couple of years now since I’ve started hosting my dot files on github along with this small shell script, dot-install, that I used to deploy files into the target machines. In one word: handy. This worked really well, the only problem being how to handle different environments.

Assuming this was a problem, this weekend I’ve decided to tackle the problem and the result was a major refactoring. The dot-install script have been completely rewritten which, luckily, will solve my problem of dealing with heterogeneous environments. That also required a few changes in the dot files themselves, the most notable example are emacs related files.

Following I will describe what changed. Interestingly enough, the whole thing became very modular and generic. To the point that others may hand-pick modules to install, or even install my whole thing without harming its own files.

As an example, using my own dot repository [the following assumes bash, or a shell capable or such]:

1
2
3
$ sh <(curl https://raw.github.com/dgvncsz0f/dot-install/stable/dot-install) \
   repo=git://github.com/dgvncsz0f/dot.git \
   bundle=dsouza@home

Notice that will wipe out and substitute your files. However, you may chose another location instead of the default, $HOME directory, to install files to:

1
2
3
4
5
$ mkdir ~/dot
$ sh <(curl https://raw.github.com/dgvncsz0f/dot-install/stable/dot-install) \
   repo=git://github.com/dgvncsz0f/dot.git \
   bundle=dsouza@home                      \
   root=~/dot

And that [should] be safe. Another interesting variation:

1
2
3
$ sh <(curl https://raw.github.com/dgvncsz0f/dot-install/stable/dot-install) \
   repo=git://github.com/dgvncsz0f/dot.git \
   module=editor/emacs

Install only the emacs files. Obviously you may combine these options at will. More information about them, and how use it you may find at:

dot-install

Previously, there were files on the git repository, in a non-structured manner. The knowledge of where files/directory gets deployed to where on the filesystem was buried deep down in the script, which is Ok though. However, there was this small inconvenient. How should I handle files that are specific to one specific machine?

The solution back then, in short, was to use local files. There were files or even scripts that would create or modify things upon install. That is a indeed a minor problem, but it felt awkward. I’m using a central repository to hold my files in the first place, hosting files locally became, to say the least, inconsistent.

To fix this, the first change was to make dot-install generic. Now, things are organized in modules and bundles [group of modules]. The real difference is that instead of the script knowing where to put each file, the module itself informs where to deploy files to.

Nothing too fancy, though, although there are things like pre & post hooks [1].

The actual details of how modules and bundles work can be found at:

As we can selectively install modules, I can now have everything available on github and create bundles for each environment that I have.

dot

In order to make use of the new script, some things have to be changed, mostly easy stuff. One module in particular that deserves mentioning is emacs. Cutting to the chase, each plugin/extension got into its own module and it is possible to let some of them out of the install.

To accomplish this I created a site-lisp directory and a site-start.d under ~/emacs.d. That will do exactly the same as the system wide directories, but it is owned by the user. The following elisp did the trick:

1
2
3
4
5
6
7
8
(let ((user-site-lisp-dir (concat user-emacs-directory "share/emacs/site-lisp/"))
      (user-theme-dir (concat user-emacs-directory "themes/"))
      (user-site-start-dir (concat user-emacs-directory "libexec/emacs/site-start.d/")))
  (add-to-list 'custom-theme-load-path user-theme-dir)
  (let ((default-directory user-site-lisp-dir))
    (normal-top-level-add-to-load-path '("."))
    (normal-top-level-add-subdirs-to-load-path))
  (mapcar 'load (file-expand-wildcards (concat user-site-start-dir "*.el"))))

What this does is 1) to add everything in the site-lisp to the load-path and 2) load all files that are in site-start.d.

A couple of examples using this layout might be found at:

Conclusion

Customizing, configuring and improving software, things that we use on a daily basis, takes lots of effort and time. It is certainly an ongoing, never ending, task. Sadly deployment of those files is often neglected.

The ability to perform one-click install is invaluable. It doesn’t really matter what you will use. It might be a tar file, a debian package or a simple shell script as I do. The important thing, things that comes to mind right now, is to be easy to replicate, upgrade, add new functionality rollback and test those files.

That will save a lot of time, certainly. We really must treat those files as code, taking into consideration the deployment part.

I don’t really hope that anyone will benefit from this other than myself, but the idea is worth sharing I suppose.

As usual, any feedback will be highly appreciated.

[1]https://github.com/dgvncsz0f/dot-install/blob/stable/README.rst#hooks

1st Commit - Revisited

This is my second attempt to publish stuff on a regular basis. Previously I’ve been using org-mode + jekyll and github. It was a great platform for publishing.

This time I’ll give a try to octopress. It is very straightforward to use, has a couple of themes available, and uses jekyll as well.

Org-mode is no longer, though. On the other hand, it has a plugin that allows me using rst.

I guess it is the new year’s eve approaching, which brings that almost unconscious desire to change something as a new year comes in. The plan this time is to write something at least once a month.

It sounds easy, again.