Ubuntu CI Engine

Merge lp:~canonical-ci-engineering/uci-engine/doc-msging into lp:uci-engine

  1. doc-msging
  2. Merge into trunk
Proposed by Francis Ginther
Status: Needs review
Proposed branch: lp:~canonical-ci-engineering/uci-engine/doc-msging
Merge into: lp:uci-engine
Diff against target: 293 lines (+251/-1)
4 files modified
docs/conf.py (+2/-1)
docs/index.rst (+2/-0)
docs/messaging.rst (+34/-0)
docs/snappy.rst (+213/-0)
To merge this branch: bzr merge lp:~canonical-ci-engineering/uci-engine/doc-msging
Reviewer Review Type Date Requested Status
Francis Ginther Needs Fixing
Paul Larson (community) Needs Fixing
Review via email: mp+246366@code.launchpad.net

Commit message

Add documentation for the snappy testing service workflow and message API.

Description of the change

A proposed solution for a multi-service workflow and message handling for the snappy image testing project.

Note:
 - This proposal requires modification to the AMQPWorker class and/or creation of a similar worker class.

To generate docs (requires graphviz package):
$ (cd docs; make html)
$ firefox docs/_build/html/messaging.html
$ firefox docs/_build/html/snappy.html

To post a comment you must log in.
Revision history for this message
Paul Larson (pwlars) wrote :

Great improvements here! a couple of comments below.

review: Needs Fixing
Reply
Revision history for this message
Vincent Ladeuil (vila) wrote :

Some answers inline while I'm preparing to push a new revision.

Reply
Revision history for this message
Ursula Junque (ursinha) wrote :

I thought we decided not to be snappy specific. So s/snappy/ubuntucore/, is there a reason to be using snappy?

Reply
Revision history for this message
Francis Ginther (fginther) wrote :

I'm working on an update to this. So as not to delay, here is the summary:

There is no need to define a workflow up front. So the snappy image watcher need not be concerned with this. Instead each service will send its output message to a default queue. For the snappy image watcher, we can begin by calling this 'snappy.images'. So the watcher would output a message on this queue that has the information on the image to test and the test branch to use (the one with the snappy dep8 tests). It may need to add other data, but we'll have to figure that out later

The snappy image watcher and every other service will be deployed to send their output messages to a default queue. However, to facilitate writing tests, the output queue can be changed by specifying a specific queue in the received message via the 'output_queue' key. This doesn't directly apply to the snappy image watcher as it has no input queue. However, it make sense to have some similar mechanism to override the output queue for writing tests (will try to put some thought into this)

review: Needs Fixing
Reply

Unmerged revisions

933. By Francis Ginther

Remove 'Workflow Definition' and replace with description using default queue names.

932. By Vincent Ladeuil

Clarify progress queue role.

931. By Vincent Ladeuil

s/test_source/test_source_branch/

930. By Vincent Ladeuil

Use the already existing uci/ namespace for glance images.

929. By Vincent Ladeuil

s/image_version/image_revision/ as it's the established word.

928. By Vincent Ladeuil

Rename image.builds queue to snappy.images.
Define snappy request.
Mention that reduced workflow can be defined.

927. By Vincent Ladeuil

Fix formatting.

926. By Francis Ginther

Add workflow definition and message examples for snappy workflow.

925. By Vincent Ladeuil

Fix typo.

924. By Vincent Ladeuil

Describes snappy components and their queues at a high level.

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
1=== modified file 'docs/conf.py'
2--- docs/conf.py 2014-10-07 10:04:01 +0000
3+++ docs/conf.py 2015-01-15 14:34:55 +0000
4@@ -27,6 +27,7 @@
5 # coming with Sphinx (named 'sphinx.ext.*') or your custom ones.
6 extensions = [
7 'sphinx.ext.autodoc',
8+ 'sphinx.ext.graphviz',
9 ]
10
11 # Add any paths that contain templates here, relative to this directory.
12@@ -43,7 +44,7 @@
13
14 # General information about the project.
15 project = u'Ubuntu CI Engine'
16-copyright = u'2013-2014, Canonical Ltd.'
17+copyright = u'2013-2015, Canonical Ltd.'
18
19 # The version info for the project you're documenting, acts as replacement for
20 # |version| and |release|, also used in various other places throughout the
21
22=== modified file 'docs/index.rst'
23--- docs/index.rst 2014-10-07 10:04:01 +0000
24+++ docs/index.rst 2015-01-15 14:34:55 +0000
25@@ -23,9 +23,11 @@
26
27 sequence
28 components
29+ messaging
30 api
31 style
32 oauth
33+ snappy
34 ppa-hooks
35 using-lxc-for-local-development
36 upgrade
37
38=== added file 'docs/messaging.rst'
39--- docs/messaging.rst 1970-01-01 00:00:00 +0000
40+++ docs/messaging.rst 2015-01-15 14:34:55 +0000
41@@ -0,0 +1,34 @@
42+========================================
43+ Exchanging messages between components
44+========================================
45+
46+Introduction
47+============
48+
49+Components can communicate via messages stored in queues. We use `rabbitmq
50+<http://www.rabbitmq.com/>`_ for the implementation.
51+
52+
53+Documentation
54+=============
55+
56+The relations between components and the queues they subscribe from/publish
57+to can be documented with ``graphviz`` as shown below:
58+
59+.. digraph:: foo
60+
61+ rankdir=LR
62+ node [shape=box]
63+ "test_runner_queue" [shape=record label="{||}" height=0.25 fixedsize=true]
64+ "test_runner_queue" -> "test runner" [label="test_runner"]
65+
66+ "test runner" -> "progress_queue"
67+ "progress_queue" -> "results" [label="<progress>"]
68+ "progress_queue" [shape=record label="{||}" height=0.25 fixedsize=true]
69+
70+In this diagram, the test runner component subscribes to the queue named
71+"test_runner" and publish progress and completion messages in the
72+"<progress>" queue. The angle brackets are used to remind that the real
73+name of the queue is set by the requester inside the request itself (as
74+opposed to a fixed name like "test_runner").
75+
76
77=== added file 'docs/snappy.rst'
78--- docs/snappy.rst 1970-01-01 00:00:00 +0000
79+++ docs/snappy.rst 2015-01-15 14:34:55 +0000
80@@ -0,0 +1,213 @@
81+===============
82+Snappy Workflow
83+===============
84+
85+
86+Introduction
87+============
88+
89+Testing snappy images and providing data to support image promotion decision
90+is implemented with a workflow defining the following steps, each step being
91+completed by a specific service:
92+
93+* Identify a new core image via new service
94+
95+* Convert core image to glance image via new service
96+
97+* Execute dep8 tests via test runner
98+
99+* Make test results available via new service
100+
101+Workflow Handling
102+=================
103+
104+The workflow is not managed and executed by a specific service, instead it is
105+executed through the default arrangement of the message queues. By default,
106+each service will read from a specific queue and write to a specific queue.
107+These queues are named when the service is deployed.
108+
109+Communication between the services is acheived by passing a message from one
110+service to the next. The message that a service receives from its input queue
111+will be the same message sent to its output queue plus any modifications and
112+additions that the service needs to make. Therefore, the message may contain
113+more fields then the current service requires. These extra fields must be
114+passed on because a downstream service may require them.
115+
116+The default output queue can be over-ridden by specifing an alternate
117+``output_queue`` in the message sent to a service. This is intended to
118+facilitate writing tests that need to send specific test messages to a service
119+and consume the output that the service creates. Otherwise, the output would
120+continue to route to the default queue which may already be consumed by a
121+running service.
122+
123+Snappy request
124+==============
125+
126+.. code::
127+
128+ message = {
129+ 'image_channel': 'ubuntu-core/devel',
130+ 'image_revision': '20150109.2',
131+ 'test_source_branch': 'lp:snappy-tests',
132+ 'test_source_revno': '10',
133+ }
134+
135+A snappy request should contain all parameters needed by the services
136+involved in the workflow.
137+
138+
139+Snappy image watcher
140+====================
141+
142+.. digraph:: foo
143+
144+ rankdir=LR
145+ node [shape=box]
146+ "snappy\nimage\nwatcher" -> "si_queue"
147+ "si_queue" [shape=record label="{||}" height=0.25 fixedsize=true]
148+ "" [style=invis]
149+ "si_queue" -> "" [label="snappy.images"]
150+
151+This service is implemented as a cron job that compares a ``system-image``
152+stream to the existing snappy images in glance.
153+
154+When a new image is detected, a request to build the corresponding glance
155+image is published to the ``snappy.images`` queue as described in `Snappy
156+Request`_.
157+
158+Output:
159+
160+.. code::
161+
162+ message = {
163+ 'image_channel': 'ubuntu-core/devel',
164+ 'image_revision': '20150109.2',
165+ 'test_source_branch': 'lp:snappy-tests',
166+ 'test_source_revno': '10',
167+ }
168+
169+
170+Snappy image provider
171+=====================
172+
173+.. digraph:: foo
174+
175+ rankdir=LR
176+ node [shape=box]
177+ "si_queue" [shape=record label="{||}" height=0.25 fixedsize=true]
178+ "si_queue" -> "snappy\nimage\nprovider" [label="snappy.images"]
179+
180+ "tr_queue" [shape=record label="{||}" height=0.25 fixedsize=true]
181+ "snappy\nimage\nprovider" -> "tr_queue"
182+
183+ "" [style=invis]
184+ "tr_queue" -> "" [label="test_runner"]
185+
186+This service subscribes to the ``snappy.images`` queue and creates a glance
187+image for the data specified in the request.
188+
189+It then posts a request to test the image in the ``test_runner`` queue.
190+
191+Input (``snappy.images``)
192+------------------------
193+
194+.. code::
195+
196+ message = {
197+ 'image_channel': 'ubuntu-core/devel',
198+ 'image_revision': '20150109.2',
199+ 'test_source_branch': 'lp:snappy-tests',
200+ 'test_source_revno': '10',
201+ }
202+
203+Output (``test_runner``)
204+------------------------
205+
206+.. code::
207+
208+ message = {
209+ 'image_channel': 'ubuntu-core/devel',
210+ 'image_revision': '20150109.2',
211+ 'test_source_branch': 'lp:snappy-tests',
212+ 'test_source_revno': '10',
213+ # Add the name of the glance image that was uploaded
214+ 'image_id': 'uci/snappy/ubuntu-core/devel/20150109.2',
215+ }
216+
217+Test runner
218+===========
219+
220+.. digraph:: foo
221+
222+ rankdir=LR
223+ node [shape=box]
224+ "tr_queue" [shape=record label="{||}" height=0.25 fixedsize=true]
225+ "tr_queue" -> "test\nrunner" [label="test_runner"]
226+
227+ "sr_queue" [shape=record label="{||}" height=0.25 fixedsize=true]
228+ "test\nrunner" -> "sr_queue"
229+
230+ "" [style=invis]
231+ "sr_queue" -> "" [label="snappy.results"]
232+
233+This services subscribes to the ``test_runner`` queue, calls ``adt-run`` for
234+the snappy selftests.
235+
236+It then publishes a request to the ``snappy.results`` queue.
237+
238+Input (``test_runner``)
239+-----------------------
240+
241+.. code::
242+
243+ message = {
244+ 'image_channel': 'ubuntu-core/devel',
245+ 'image_revision': '20150109.2',
246+ 'test_source_branch': 'lp:snappy-tests',
247+ 'test_source_revno': '10',
248+ 'image_id': 'uci/snappy/ubuntu-core/devel/20150109.2',
249+ }
250+
251+Output (``snappy.results``)
252+---------------------------
253+
254+.. code::
255+
256+ message = {
257+ 'image_channel': 'ubuntu-core/devel',
258+ 'image_revision': '20150109.2',
259+ 'test_source_branch': 'lp:snappy-tests',
260+ 'test_source_revno': '10',
261+ 'image_id': 'uci/snappy/ubuntu-core/devel/20150109.2',
262+ # Add the artifacts produced by the test_runner
263+ 'artifacts': ['swift-reference-to-tarball'],
264+ }
265+
266+Snappy results provider
267+=======================
268+
269+.. digraph:: foo
270+
271+ rankdir=LR
272+ node [shape=box]
273+ "sr_queue" [shape=record label="{||}" height=0.25 fixedsize=true]
274+ "sr_queue" -> "snappy\nresults" [label="snappy.results"]
275+
276+This service subscribes to the ``snappy.results`` queue and transforms the
277+test runner results into a human readable format for snappy image promotion
278+decision.
279+
280+Input (``snappy.results``)
281+--------------------------
282+
283+.. code::
284+
285+ message = {
286+ 'image_channel': 'ubuntu-core/devel',
287+ 'image_revision': '20150109.2',
288+ 'image_id': 'uci/snappy/ubuntu-core/devel/20150109.2',
289+ 'test_source_branch': 'lp:snappy-tests',
290+ 'test_source_revno': '10',
291+ 'artifacts': ['swift-reference-to-tarball'],
292+ }
293+

Subscribers

People subscribed via source and target branches

to all changes: