<?xml version="1.0"?><!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
"http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd"><html xmlns="http://www.w3.org/1999/xhtml" lang="en"><head><title>Twisted Documentation: Asynchronous Programming with Twisted</title><link href="../howto/stylesheet.css" type="text/css" rel="stylesheet" /></head><body bgcolor="white"><h1 class="title">Asynchronous Programming with Twisted</h1><div class="toc"><ol><li><a href="#auto0">Introduction to concurrent programming</a></li><ul><li><a href="#auto1">Waiting for answers</a></li><li><a href="#auto2">Not waiting on data</a></li><li><a href="#auto3">Non-blocking calls</a></li><li><a href="#auto4">Callbacks</a></li></ul><li><a href="#auto5">Deferreds</a></li><li><a href="#auto6">The Problem that Deferreds Solve</a></li><li><a href="#auto7">Deferreds - a signal that data is yet to come</a></li><ul><li><a href="#auto8">Callbacks</a></li><li><a href="#auto9">Error handling: errbacks</a></li></ul><li><a href="#auto10">Conclusion</a></li><ul><li><a href="#auto11">See also</a></li></ul></ol></div><div class="content"><span></span><p>This document is a introduction to the asynchronous programming model, and
to Twisted's Deferred abstraction, which symbolises a 'promised' result and
which can pass an eventual result to handler functions.</p><p>This document is for readers new to Twisted who are familiar with the
Python programming language and, at least conceptually, with core networking
conepts such as servers, clients and sockets. This document will give you a
high level overview of concurrent programming (interleaving several tasks) and
of Twisted's concurrency model: <strong>non-blocking code</strong> or
<strong>asynchronous code</strong>.</p><p>After discussing the concurrency model of which Deferreds are a part, it
will introduce the methods of handling results when a function returns a
Deferred object.</p><h2>Introduction to concurrent programming<a name="auto0"></a></h2><p>
Many computing tasks take some time to complete, and there are two reasons
why a task might take some time:
</p><ol><li>it is computationally intensive (for example factorising large numbers)
and requires a certain amount of CPU time to calculate the answer; or</li><li>it is not computationally intensive but has to wait for data to be
available to produce a result.</li></ol><h3>Waiting for answers<a name="auto1"></a></h3><p>A fundamental feature of network programming is that of waiting for data.
Imagine you have a function which sends an email summarising some information.
This function needs to connect to a remote server, wait for the remote server
to reply, check that the remote server can process the email, wait for the
reply, send the email, wait for the confirmation, and then disconnect.</p><p>Any one of these steps may take a long period of time. Your program might
use the simplest of all possible models, in which it actually sits and waits
for data to be sent and received, but in this case it has some very obvious
and basic limitations: it can't send many emails at once; and in fact it can't
do anything else while it is sending an email.</p><p>Hence, all but the simplest network programs avoid this model. You can use
one of several different models to allow your program to keep doing whatever
tasks it has on hand while it is waiting for something to happen before a
particular task can continue.</p><h3>Not waiting on data<a name="auto2"></a></h3><p>There are many ways to write network programs. The main ones are:</p><ol><li>handle each connection in a separate operating system process, in
which case the operating system will take care of letting other processes
run while one is waiting;</li><li>handle each connection in a separate thread<a href="#footnote-1" title="There are variations on this method, such as a limited-size pool of threads servicing all connections, which are essentially just optimizations of the same idea."><super>1</super></a> in which the
threading framework takes care of letting other threads run while one is
waiting; or</li><li>use non-blocking system calls to handle all connections
in one thread.</li></ol><h3>Non-blocking calls<a name="auto3"></a></h3><p>The normal model when using the Twisted framework is the third model:
non-blocking calls.</p><p>When dealing with many connections in one thread, the scheduling is the
responsibility of the application, not the operating system, and is usually
implemented by calling a registered function when each connection is ready to
for reading or writing -- commonly known as <strong>asynchronous</strong>,
<strong>event-driven</strong> or <strong>callback-based</strong>
programming.</p><p>In this model, the earlier email sending function would work something
like this:</p><ol><li>it calls a connection function to connect to the remote server;</li><li>the connection function returns immediately, with the implication that
the notify the email sending library will be called when the connect has
been made; and</li><li>once the connection is made, the connect mechanism notifies the email
sending function that the connection is ready.</li></ol><p>What advantage does the above sequence have over our original blocking
sequence? The advantage is that while the email sending function can't do the
next part of its job until the connection is open, the rest of the program can
do other tasks, like begin the opening sequence for other email connections.
Hence, the entire program is not waiting for the connection.</p><h3>Callbacks<a name="auto4"></a></h3><p>The typical asynchronous model for alerting an application that some data
is ready for it is known as a <em>callback</em>. The application calls a
function to request some data, and in this call, it also passes a
callback function that should be called when the data is ready with the data
as an argument. The callback function should therefore perform whatever tasks
it was that the application needed that data for.</p><p>In synchonous programming, a function requests data, waits for the data,
and then processes it. In asynchronous programming, a function requests the
data, and lets the library call the callback function when the data is
ready.</p><a name="deferreds"></a><h2>Deferreds<a name="auto5"></a></h2><p>Twisted uses the <code base="twisted.internet.defer" class="API">Deferred</code> object to manage the callback
sequence. The client application attaches a series of functions to the
deferred to be called in order when the results of the asychronous request are
available (this series of functions is known as a series of
<strong>callbacks</strong>, or a <strong>callback chain</strong>), together
with a series of functions to be called if there is an error in the
asychronous request (known as a series of <strong>errbacks</strong> or an
<strong>errback chain</strong>). The asychronous library code calls the first
callback when the result is available, or the first errback when an error
occurs, and the <code>Deferred</code> object then hands the results of each
callback or errback function to the next function in the chain.</p><h2>The Problem that Deferreds Solve<a name="auto6"></a></h2><p> It is the second class of concurrency problem — non-computationally
intensive tasks that involve an appreciable delay — that Deferreds are
designed to help solve. Functions that wait on hard drive access, database
access, and network access all fall into this class, although the time delay
varies. </p><p> Deferreds are designed to enable Twisted programs to wait for data
without hanging until that data arrives. They do this by giving a
simple management interface for callbacks to libraries and
applications. Libraries know that they always make their results
available by calling <code base="twisted.internet.defer" class="API">Deferred.callback</code> and errors by
calling
<code base="twisted.internet.defer" class="API">Deferred.errback</code>. Applications set up
result handlers by attaching callbacks and errbacks to deferreds in the order
they want them called.</p><p> The basic idea behind Deferreds, and other solutions to this problem, is
to keep the CPU as active as possible. If one task is waiting on data, rather
than have the CPU (and the program!) idle waiting for that data (a process
normally called "blocking"), the program performs other operations
in the meantime, and waits for some signal that data is ready to be processed
before returning to that process. </p><p>
In Twisted, a function signals to the calling function that it is waiting by
returning a Deferred. When the data is available, the program activates
the callbacks on that Deferred to process the data.
</p><h2>Deferreds - a signal that data is yet to come<a name="auto7"></a></h2><p>In our email sending example above, a parent function calls a function to
connect to the remote server. Asynchrony requires that this connection
function return <em>without waiting for the result</em> so that the parent
function can do other things. So how does the parent function or its
controlling program know that the connection doesn't exist yet, and how does
it use the connection once it does exist?</p><p>Twisted has an object that signals this situation. When the connection
function returns, it signals that the operation is incomplete by returning a
<code class="API">twisted.internet.defer.Deferred</code> object.</p><p>The Deferred has two purposes. The first is that it says "I am a
signal that the result of whatever you wanted me to do is still pending."
The second is that you can ask the Deferred to run things when the data
does arrive.</p><h3>Callbacks<a name="auto8"></a></h3><p>The way you tell a Deferred what to do with the data once it arrives is by
adding a callback — asking the Deferred to call a function once the data
arrives.</p><p>One Twisted library function that returns a Deferred is <code class="API">twisted.web.client.getPage</code>. In this example, we call
<code>getPage</code>, which returns a Deferred, and we attach a callback to
handle the contents of the page once the data is available:</p><pre class="python">
<span class="py-src-keyword">from</span> <span class="py-src-variable">twisted</span>.<span class="py-src-variable">web</span>.<span class="py-src-variable">client</span> <span class="py-src-keyword">import</span> <span class="py-src-variable">getPage</span>
<span class="py-src-keyword">from</span> <span class="py-src-variable">twisted</span>.<span class="py-src-variable">internet</span> <span class="py-src-keyword">import</span> <span class="py-src-variable">reactor</span>
<span class="py-src-keyword">def</span> <span class="py-src-identifier">printContents</span>(<span class="py-src-parameter">contents</span>):
<span class="py-src-string">'''
This is the 'callback' function, added to the Deferred and called by
it when the promised data is available
'''</span>
<span class="py-src-keyword">print</span> <span class="py-src-string">"The Deferred has called printContents with the following contents:"</span>
<span class="py-src-keyword">print</span> <span class="py-src-variable">contents</span>
<span class="py-src-comment"># Stop the Twisted event handling system -- this is usually handled
</span> <span class="py-src-comment"># in higher level ways
</span> <span class="py-src-variable">reactor</span>.<span class="py-src-variable">stop</span>()
<span class="py-src-comment"># call getPage, which returns immediately with a Deferred, promising to
</span><span class="py-src-comment"># pass the page contents onto our callbacks when the contents are available
</span><span class="py-src-variable">deferred</span> = <span class="py-src-variable">getPage</span>(<span class="py-src-string">'http://twistedmatrix.com/'</span>)
<span class="py-src-comment"># add a callback to the deferred -- request that it run printContents when
</span><span class="py-src-comment"># the page content has been downloaded
</span><span class="py-src-variable">deferred</span>.<span class="py-src-variable">addCallback</span>(<span class="py-src-variable">printContents</span>)
<span class="py-src-comment"># Begin the Twisted event handling system to manage the process -- again this
</span><span class="py-src-comment"># isn't the usual way to do this
</span><span class="py-src-variable">reactor</span>.<span class="py-src-variable">run</span>()
</pre><p>A very common use of Deferreds is to attach two callbacks. The result of the
first callback is passed to the second callback:</p><pre class="python">
<span class="py-src-keyword">from</span> <span class="py-src-variable">twisted</span>.<span class="py-src-variable">web</span>.<span class="py-src-variable">client</span> <span class="py-src-keyword">import</span> <span class="py-src-variable">getPage</span>
<span class="py-src-keyword">from</span> <span class="py-src-variable">twisted</span>.<span class="py-src-variable">internet</span> <span class="py-src-keyword">import</span> <span class="py-src-variable">reactor</span>
<span class="py-src-keyword">def</span> <span class="py-src-identifier">lowerCaseContents</span>(<span class="py-src-parameter">contents</span>):
<span class="py-src-string">'''
This is a 'callback' function, added to the Deferred and called by
it when the promised data is available. It converts all the data to
lower case
'''</span>
<span class="py-src-keyword">return</span> <span class="py-src-variable">contents</span>.<span class="py-src-variable">lower</span>()
<span class="py-src-keyword">def</span> <span class="py-src-identifier">printContents</span>(<span class="py-src-parameter">contents</span>):
<span class="py-src-string">'''
This a 'callback' function, added to the Deferred after lowerCaseContents
and called by it with the results of lowerCaseContents
'''</span>
<span class="py-src-keyword">print</span> <span class="py-src-variable">contents</span>
<span class="py-src-variable">reactor</span>.<span class="py-src-variable">stop</span>()
<span class="py-src-variable">deferred</span> = <span class="py-src-variable">getPage</span>(<span class="py-src-string">'http://twistedmatrix.com/'</span>)
<span class="py-src-comment"># add two callbacks to the deferred -- request that it run lowerCaseContents
</span><span class="py-src-comment"># when the page content has been downloaded, and then run printContents with
</span><span class="py-src-comment"># the result of lowerCaseContents
</span><span class="py-src-variable">deferred</span>.<span class="py-src-variable">addCallback</span>(<span class="py-src-variable">lowerCaseContents</span>)
<span class="py-src-variable">deferred</span>.<span class="py-src-variable">addCallback</span>(<span class="py-src-variable">printContents</span>)
<span class="py-src-variable">reactor</span>.<span class="py-src-variable">run</span>()
</pre><h3>Error handling: errbacks<a name="auto9"></a></h3><p>Just as a asynchronous function returns before its result is available, it
may also return before it is possible to detect errors: failed connections,
erroneous data, protocol errors, and so on. Just as you can add callbacks to a
Deferred which it calls when the data you are expecting is available, you can
add error handlers ('errbacks') to a Deferred for it to call when an error
occurs and it cannot obtain the data:</p><pre class="python">
<span class="py-src-keyword">from</span> <span class="py-src-variable">twisted</span>.<span class="py-src-variable">web</span>.<span class="py-src-variable">client</span> <span class="py-src-keyword">import</span> <span class="py-src-variable">getPage</span>
<span class="py-src-keyword">from</span> <span class="py-src-variable">twisted</span>.<span class="py-src-variable">internet</span> <span class="py-src-keyword">import</span> <span class="py-src-variable">reactor</span>
<span class="py-src-keyword">def</span> <span class="py-src-identifier">errorHandler</span>(<span class="py-src-parameter">error</span>):
<span class="py-src-string">'''
This is an 'errback' function, added to the Deferred which will call
it in the event of an error
'''</span>
<span class="py-src-comment"># this isn't a very effective handling of the error, we just print it out:
</span> <span class="py-src-keyword">print</span> <span class="py-src-string">"An error has occurred: <%s>"</span> % <span class="py-src-variable">str</span>(<span class="py-src-variable">error</span>)
<span class="py-src-comment"># and then we stop the entire process:
</span> <span class="py-src-variable">reactor</span>.<span class="py-src-variable">stop</span>()
<span class="py-src-keyword">def</span> <span class="py-src-identifier">printContents</span>(<span class="py-src-parameter">contents</span>):
<span class="py-src-string">'''
This a 'callback' function, added to the Deferred and called by it with
the page content
'''</span>
<span class="py-src-keyword">print</span> <span class="py-src-variable">contents</span>
<span class="py-src-variable">reactor</span>.<span class="py-src-variable">stop</span>()
<span class="py-src-comment"># We request a page which doesn't exist in order to demonstrate the
</span><span class="py-src-comment"># error chain
</span><span class="py-src-variable">deferred</span> = <span class="py-src-variable">getPage</span>(<span class="py-src-string">'http://twistedmatrix.com/does-not-exist'</span>)
<span class="py-src-comment"># add the callback to the Deferred to handle the page content
</span><span class="py-src-variable">deferred</span>.<span class="py-src-variable">addCallback</span>(<span class="py-src-variable">printContents</span>)
<span class="py-src-comment"># add the errback to the Deferred to handle any errors
</span><span class="py-src-variable">deferred</span>.<span class="py-src-variable">addErrback</span>(<span class="py-src-variable">errorHandler</span>)
<span class="py-src-variable">reactor</span>.<span class="py-src-variable">run</span>()
</pre><h2>Conclusion<a name="auto10"></a></h2><p>In this document, you have:</p><ol><li>seen why non-trivial network programs need to have some form of concurrency;</li><li>learnt that the Twisted framework supports concurrency in the form of
asynchronous calls;</li><li>learnt that the Twisted framework has Deferred objects that manage callback
chains;</li><li>seen how the <code base="twisted.web.client" class="API">getPage</code>
function returns a Deferred object;</li><li>attached callbacks and errbacks to that Deferred; and</li><li>seen the Deferred's callback chain and errback chain fire.</li></ol><h3>See also<a name="auto11"></a></h3><p>Since the Deferred abstraction is such a core part of programming with
Twisted, there are several other detailed guides to it:</p><ol><li><a href="defer.html">Using Deferreds</a>, a more complete guide to
using Deferreds, including Deferred chaining.</li><li><a href="gendefer.html">Generating Deferreds</a>, a guide to creating
Deferreds and firing their callback chains.</li></ol><h2>Footnotes</h2><ol><li><a name="footnote-1"><span xmlns="http://www.w3.org/1999/xhtml" class="footnote">There
are variations on this method, such
as a limited-size pool of threads servicing all connections, which are
essentially just optimizations of the same idea.</span></a></li></ol></div><p><a href="../howto/index.html">Index</a></p><span class="version">Version: 2.5.0</span></body></html>
distrib
>
Mandriva
>
2008.1
>
x86_64
>
media
>
main-release
>
by-pkgid
>
db93d7191b12a3d5ce887da46fc79bf1
>
files
>
96
