Ubuntu Test Cases

Merge lp:~canonical-platform-qa/ubuntu-test-cases/dep8-app-startup into lp:ubuntu-test-cases/touch

  1. dep8-app-startup
  2. Merge into touch
Proposed by Leonardo Arias Fonseca
Status: Needs review
Proposed branch: lp:~canonical-platform-qa/ubuntu-test-cases/dep8-app-startup
Merge into: lp:ubuntu-test-cases/touch
Diff against target: 461 lines (+424/-0)
7 files modified
tests/app-startup/README.md (+37/-0)
tests/app-startup/debian/changelog (+5/-0)
tests/app-startup/debian/tests/app-startup (+18/-0)
tests/app-startup/debian/tests/app-startup-nfss-datafile-generator.py (+175/-0)
tests/app-startup/debian/tests/app-startup-root-setup (+33/-0)
tests/app-startup/debian/tests/control (+11/-0)
tests/app-startup/debian/tests/test_app_startup.py (+145/-0)
To merge this branch: bzr merge lp:~canonical-platform-qa/ubuntu-test-cases/dep8-app-startup
Reviewer Review Type Date Requested Status
Allan LeSage (community) Needs Fixing
Review via email: mp+241613@code.launchpad.net

This proposal supersedes a proposal from 2014-11-12.

Commit message

Added dep8 tests for app startup.

Description of the change

There is room for a lot of improvements, but the goal of this branch was to copy the old utah test into dep8. More changes will come later.

To post a comment you must log in.
Revision history for this message
Christopher Lee (veebers) wrote :

Looking good so far.

Reply
lp:~canonical-platform-qa/ubuntu-test-cases/dep8-app-startup updated
365. By Leonardo Arias Fonseca

Fixed the README command.

366. By Leonardo Arias Fonseca

Fixed the README command.

367. By Leonardo Arias Fonseca

Moved the setup to a test.

368. By Leonardo Arias Fonseca

Fixed the call to datetime.

Revision history for this message
Ted Gould (ted) wrote :

You won't get the start message callback events doing this because you haven't send the environment variable before unity starts. Which is required for the version of libual that's loaded into its memory region. That tracepoint is important to know how long Unity is delaying the startup of apps.

Reply
Revision history for this message
Allan LeSage (allanlesage) wrote :

I'm seeing that babeltrace isn't available on the phone image? http://pastebin.ubuntu.com/9009103/

review: Needs Fixing
Reply
Revision history for this message
Leonardo Arias Fonseca (elopio) wrote :

Allan, that's weird. This is from my krillin:

phablet@ubuntu-phablet:~$ apt-cache search babeltrace
libbabeltrace-ctf-dev - Common Trace Format (CTF) development files
libbabeltrace-ctf1 - Common Trace Format (CTF) library
libbabeltrace-dev - Babeltrace development files
libbabeltrace1 - Babeltrace conversion libraries
babeltrace - Trace conversion program
python3-babeltrace - Babeltrace conversion libraries

In what phone and version are you running it?

Reply
Revision history for this message
Allan LeSage (allanlesage) wrote :

I'm on mako, maybe that explains?

phablet@ubuntu-phablet:~$ apt-cache search babeltrace
phablet@ubuntu-phablet:~$

Reply
Revision history for this message
Leonardo Arias Fonseca (elopio) wrote :

> You won't get the start message callback events doing this because you haven't
> send the environment variable before unity starts. Which is required for the
> version of libual that's loaded into its memory region. That tracepoint is
> important to know how long Unity is delaying the startup of apps.

<elopio> ted: thanks for looking at it. But didn't you tell me that if after writing the config file I rebooted the machine, the right env vars will be set?
<ted> elopio, Ah, okay. Was looking at app-startup. Why is that script setting them?
<elopio> ted: I'm not sure. I thought something might read the env vars and not the initctl vars.
<elopio> maybe it's not needed and I can remove it.

Reply

Unmerged revisions

368. By Leonardo Arias Fonseca

Fixed the call to datetime.

367. By Leonardo Arias Fonseca

Moved the setup to a test.

366. By Leonardo Arias Fonseca

Fixed the README command.

365. By Leonardo Arias Fonseca

Fixed the README command.

364. By Leonardo Arias Fonseca

Copied the generator script from veebers.

363. By Leonardo Arias Fonseca

Added a backlist.

362. By Leonardo Arias Fonseca

Added a timetout to the app launch.

361. By Leonardo Arias Fonseca

Print the unittests results to stdout.

360. By Leonardo Arias Fonseca

Moved the root commands to setup.

359. By Leonardo Arias Fonseca

Checkpoing for the debian packages working.

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
1=== added directory 'tests/app-startup'
2=== added file 'tests/app-startup/README.md'
3--- tests/app-startup/README.md 1970-01-01 00:00:00 +0000
4+++ tests/app-startup/README.md 2014-11-13 18:49:53 +0000
5@@ -0,0 +1,37 @@
6+# Ubuntu applications startup test
7+
8+This test launches the applications and measure the time it takes for them to
9+be fully opened, by listening to the lttng traces.
10+
11+## Resources
12+
13+This test should be run on an Ubuntu Touch device.
14+
15+## To run the test
16+
17+1. Flash an Ubuntu Touch device.
18+2. Enable adb access on the device.
19+3. Get the test code:
20+
21+ $ bzr branch lp:ubuntu-test-cases/touch
22+
23+4. Connect the device to the runner machine with an USB cable.
24+5. Run the tests with adt-run:
25+
26+ $ adt-run -B --built-tree=touch/tests/app-startup --setup-commands=touch/tests/app-startup/debian/tests/setup --output-dir=output --- ssh -s adb -- -p <device password>
27+
28+## Test results
29+
30+The test executable exits with 0 if successful, non-zero otherwise.
31+
32+## Non-functional stats data
33+
34+The NFSS json data file will be stored at the output/artifacts/ directory.
35+
36+## Triggering
37+
38+This test should be run every time there is a new Ubuntu Touch image.
39+
40+## Monitoring
41+
42+work in progress.
43
44=== added directory 'tests/app-startup/debian'
45=== added file 'tests/app-startup/debian/changelog'
46--- tests/app-startup/debian/changelog 1970-01-01 00:00:00 +0000
47+++ tests/app-startup/debian/changelog 2014-11-13 18:49:53 +0000
48@@ -0,0 +1,5 @@
49+app-startup-tests (0.1-1) unstable; urgency=low
50+
51+ * Initial release.
52+
53+ -- Leonardo Arias Fonseca <leo.arias@canonical.com> Wed, 12 Nov 2014 09:44:55 -0600
54
55=== added directory 'tests/app-startup/debian/tests'
56=== added file 'tests/app-startup/debian/tests/app-startup'
57--- tests/app-startup/debian/tests/app-startup 1970-01-01 00:00:00 +0000
58+++ tests/app-startup/debian/tests/app-startup 2014-11-13 18:49:53 +0000
59@@ -0,0 +1,18 @@
60+#!/bin/sh
61+
62+
63+# XXX ask why is there a first run error. --elopio - 2014-11-12
64+echo "Creating dummy session ignoring the error to get around first run error."
65+
66+lttng create dummy 2> /dev/null || echo 'error'
67+lttng destroy dummy
68+
69+echo "Setting environment variables for tracing."
70+export LTTNG_UST_REGISTER_TIMEOUT=-1
71+export UBUNTU_APP_LAUNCH_LTTNG_ENABLED=TRUE
72+
73+# Run the tests and generate the lttng files.
74+python3 debian/tests/test_app_startup.py
75+
76+# Parse the lttng files and save the restuls to the nfss data file.
77+python3 debian/tests/app-startup-nfss-datafile-generator.py $ADT_ARTIFACTS/lttng_traces $ADT_ARTIFACTS
78
79=== added file 'tests/app-startup/debian/tests/app-startup-nfss-datafile-generator.py'
80--- tests/app-startup/debian/tests/app-startup-nfss-datafile-generator.py 1970-01-01 00:00:00 +0000
81+++ tests/app-startup/debian/tests/app-startup-nfss-datafile-generator.py 2014-11-13 18:49:53 +0000
82@@ -0,0 +1,175 @@
83+#!/usr/bin/env python3
84+
85+# Ubuntu Test Cases
86+# Copyright 2014 Canonical Ltd.
87+#
88+# This program is free software: you can redistribute it and/or modify it
89+# under the terms of the GNU Affero General Public License version 3, as
90+# published by the Free Software Foundation.
91+#
92+# This program is distributed in the hope that it will be useful, but
93+# WITHOUT ANY WARRANTY; without even the implied warranties of
94+# MERCHANTABILITY, SATISFACTORY QUALITY, or FITNESS FOR A PARTICULAR
95+# PURPOSE. See the GNU Affero General Public License for more details.
96+#
97+# You should have received a copy of the GNU Affero General Public License
98+# along with this program. If not, see <http://www.gnu.org/licenses/>.
99+
100+"""Export application trace events for app startup."""
101+
102+import datetime
103+import json
104+import os
105+import re
106+import subprocess
107+import sys
108+
109+import babeltrace
110+
111+
112+def usage():
113+ print(
114+ 'Usage:\n {} <source directory> <destination directory>'.format(
115+ sys.argv[0]
116+ )
117+ )
118+
119+
120+def main(source_directory, destination_directory):
121+ """source_directory contains all the resulting directories from the test
122+ run (i.e. com.ubuntu.calculator-startup-20141030-021839)
123+
124+ destination_directory is where the resulting nfss-data.json file will be
125+ written.
126+
127+ """
128+ all_applications_data = []
129+ for directory_name in os.listdir(source_directory):
130+ full_directory_path = os.path.abspath(
131+ os.path.join(source_directory , directory_name)
132+ )
133+ package_name = _get_package_name(directory_name)
134+ app_id = _get_app_id(full_directory_path)
135+
136+ traces_path = os.path.join(
137+ full_directory_path, app_id, 'ust/uid/32011/32-bit')
138+ tracepoints = parse_traces(traces_path),
139+ doc = generate_doc(tracepoints, app_id, package_name)
140+ all_applications_data.append(dict(
141+ project='app-startup',
142+ test=package_name,
143+ data=doc
144+ ))
145+
146+ if device_is_krillin():
147+ data_file_name = 'private-nfss-data.json'
148+ else:
149+ data_file_name = 'nfss-data.json'
150+
151+ dest_filepath = os.path.join(destination_directory, data_file_name)
152+ print("Writing results to %s" % (dest_filepath))
153+ with open(dest_filepath, "w") as f:
154+ json.dump(all_applications_data, f)
155+
156+
157+def parse_traces(trace_path):
158+ """Convert binary traces to dict of tracepoint names and timestamps.
159+
160+ Timestamp format is: epoch secs + nanosecs.
161+
162+ """
163+ # Tracepoints to include for app startup
164+ tracepoints = {
165+ 'libual_start': {},
166+ 'libual_start_message_sent': {},
167+ 'libual_start_message_callback': {},
168+ 'handshake_wait': {},
169+ 'handshake_complete': {},
170+ 'exec_pre_exec': {},
171+ }
172+
173+ # Build up the traces collection
174+ traces = babeltrace.TraceCollection()
175+ ret = traces.add_trace(trace_path, 'ctf')
176+ if ret is None:
177+ raise IOError('Error adding trace')
178+
179+ # Record the timestamps
180+ timestamps = {}
181+ for event in traces.events:
182+ name = event.name.split(':')[1]
183+ # We only want the event if it's the first occurance to
184+ # handle possibly messy traces.
185+ if name not in timestamps:
186+ # Keep only the timestamps we care about by event name
187+ if name in tracepoints:
188+ if len(tracepoints[name]) == 0:
189+ timestamps[name] = event.timestamp
190+ else:
191+ fieldname = name
192+ for field in tracepoints[name]:
193+ if not event[field] == tracepoints[name][field]:
194+ fieldname = None
195+ break
196+ else:
197+ fieldname += ":{}={}".format(
198+ field, tracepoints[name][field])
199+
200+ if fieldname is not None and fieldname not in timestamps:
201+ timestamps[fieldname] = event.timestamp
202+
203+ return timestamps
204+
205+
206+def generate_doc(tracepoints, app_id, package_name):
207+ return dict(
208+ package_name=package_name,
209+ app_id=app_id,
210+ android_serial=os.environ.get('ANDROID_SERIAL'),
211+ image_type=os.environ.get('IMAGE_TYPE'),
212+ build_num=os.environ.get('BUILD_NUMBER'),
213+ build_id=os.environ.get('BUILD_ID'),
214+ job_name=os.environ.get('JOB_NAME'),
215+ node_name=os.environ.get('NODE_NAME'),
216+ datetime=datetime.datetime.utcnow().isoformat(),
217+ tracepoints=tracepoints,
218+ )
219+
220+
221+def _get_package_name(directory):
222+ # com.ubuntu.calculator-startup-20141030-021839 -> com.ubuntu.calculator
223+ directory_name = os.path.basename(directory)
224+ package_name_search = re.search(
225+ '(.*)-startup.*',
226+ directory_name
227+ )
228+ try:
229+ return package_name_search.group(1)
230+ except IndexError:
231+ return "Unknown Package"
232+
233+
234+def _get_app_id(directory):
235+ return os.listdir(directory)[0]
236+
237+
238+def device_is_krillin():
239+ """Returns True if the device executing this code is a Krillin."""
240+ try:
241+ prop_output = subprocess.check_output(
242+ ['getprop', 'ro.product.device'],
243+ universal_newlines=True
244+ ).strip('\n')
245+ return prop_output == 'krillin'
246+ except FileNotFoundError:
247+ return False
248+
249+
250+if __name__ == '__main__':
251+ try:
252+ source_directory, destination_directory = sys.argv[1:]
253+ except (IndexError, ValueError):
254+ usage()
255+ exit(1)
256+
257+ main(source_directory, destination_directory)
258
259=== added file 'tests/app-startup/debian/tests/app-startup-root-setup'
260--- tests/app-startup/debian/tests/app-startup-root-setup 1970-01-01 00:00:00 +0000
261+++ tests/app-startup/debian/tests/app-startup-root-setup 2014-11-13 18:49:53 +0000
262@@ -0,0 +1,33 @@
263+#!/bin/sh
264+
265+set -e
266+
267+case "$ADT_REBOOT_MARK" in
268+ "") echo "Beginning test setup.";
269+
270+echo "Allowing LTTng events from confined applications."
271+sed 's/deny \(.*shm\/lttng-ust-.*\)/owner @{HOME}\/.lttng\/ rw,\nowner @{HOME}\/.lttng\/\* rwk,\n\/run\/lttng\/ rw,\n\/run\/lttng\/** rwk,/' -i /usr/share/apparmor/easyprof/templates/ubuntu/*/ubuntu-*
272+
273+echo "Regenerating AppArmor profiles for installed click applications."
274+aa-clickhook -f
275+
276+echo "Setting environment variables for session."
277+mkdir -p /home/$SUDO_USER/.config/upstart
278+cat > /home/$SUDO_USER/.config/upstart/lttng-trace.conf <<EOF
279+start on starting dbus
280+script
281+initctl set-env --global LTTNG_UST_REGISTER_TIMEOUT=-1
282+initctl set-env --global UBUNTU_APP_LAUNCH_LTTNG_ENABLED=TRUE
283+end script
284+EOF
285+
286+chown $SUDO_USER:$SUDO_USER /home/$SUDO_USER/.config/upstart/lttng-trace.conf
287+
288+echo "Rebooting."
289+/tmp/autopkgtest-reboot rebooted ;;
290+
291+ rebooted) echo "Rebooted." ;;
292+
293+esac
294+
295+echo "Setup ended."
296
297=== added file 'tests/app-startup/debian/tests/control'
298--- tests/app-startup/debian/tests/control 1970-01-01 00:00:00 +0000
299+++ tests/app-startup/debian/tests/control 2014-11-13 18:49:53 +0000
300@@ -0,0 +1,11 @@
301+Tests: app-startup-root-setup
302+Restrictions: needs-root
303+
304+Tests: app-startup
305+Depends: babeltrace,
306+ lttng-tools,
307+ python3-apt,
308+ python3-babeltrace,
309+ python3-fixtures,
310+ python3-testscenarios,
311+ python3-testtools
312
313=== added file 'tests/app-startup/debian/tests/test_app_startup.py'
314--- tests/app-startup/debian/tests/test_app_startup.py 1970-01-01 00:00:00 +0000
315+++ tests/app-startup/debian/tests/test_app_startup.py 2014-11-13 18:49:53 +0000
316@@ -0,0 +1,145 @@
317+#!/usr/bin/env python3
318+
319+# Ubuntu Test Cases
320+# Copyright 2014 Canonical Ltd.
321+#
322+# This program is free software: you can redistribute it and/or modify it
323+# under the terms of the GNU Affero General Public License version 3, as
324+# published by the Free Software Foundation.
325+#
326+# This program is distributed in the hope that it will be useful, but
327+# WITHOUT ANY WARRANTY; without even the implied warranties of
328+# MERCHANTABILITY, SATISFACTORY QUALITY, or FITNESS FOR A PARTICULAR
329+# PURPOSE. See the GNU Affero General Public License for more details.
330+#
331+# You should have received a copy of the GNU Affero General Public License
332+# along with this program. If not, see <http://www.gnu.org/licenses/>.
333+
334+import datetime
335+import os
336+import subprocess
337+import sys
338+import time
339+import unittest
340+
341+import apt
342+import fixtures
343+import testscenarios
344+import testtools
345+from gi.repository import Click
346+
347+
348+debian_packages = [
349+ 'address-book-app', 'dialer-app', 'messaging-app',
350+ 'ubuntu-system-settings', 'webbrowser-app'
351+]
352+black_list = [
353+ 'com.canonical.payui', 'com.ubuntu.reminders',
354+ 'com.ubuntu.scopes.youtube'
355+]
356+
357+
358+def get_click_packages():
359+ click_db = Click.DB()
360+ click_db.read()
361+ registry = Click.User.for_user(click_db, name=os.environ.get('USER'))
362+ return registry.get_package_names()
363+
364+
365+class LTTngStarted(fixtures.Fixture):
366+
367+ def __init__(self, name, output_directory):
368+ super().__init__()
369+ self.name = name
370+ self.output_directory = output_directory
371+
372+ def setUp(self):
373+ super().setUp()
374+ self.addCleanup(self._destroy_tracing_session)
375+ self._create_tracing_session()
376+ self._enable_all_userspace_events()
377+ self.addCleanup(self._stop_tracing)
378+ self._start_tracing()
379+
380+ def _create_tracing_session(self):
381+ subprocess.check_call(
382+ ['lttng', 'create', self.name, '-o', self.output_directory])
383+
384+ def _destroy_tracing_session(self):
385+ subprocess.check_call(['lttng', 'destroy', self.name])
386+
387+ def _enable_all_userspace_events(self):
388+ subprocess.check_call(['lttng', 'enable-event', '-u', '-a'])
389+
390+ def _start_tracing(self):
391+ subprocess.check_call(['lttng', 'start'])
392+
393+ def _stop_tracing(self):
394+ subprocess.check_call(['lttng', 'stop'])
395+
396+
397+class AppStartupTestCase(testscenarios.TestWithScenarios, testtools.TestCase):
398+
399+ click_packages = get_click_packages()
400+ scenarios = [
401+ ('Startup test for {}'.format(package_name), {
402+ 'package_name': package_name
403+ })
404+ for package_name in click_packages + debian_packages
405+ if package_name not in black_list
406+ ]
407+
408+ def get_app_id(self, package_name):
409+ if package_name in debian_packages:
410+ version = self.get_debian_package_version(package_name)
411+ return '{}_{}'.format(package_name, version)
412+ else:
413+ return self.get_click_app_id(package_name)
414+
415+ def get_debian_package_version(self, package_name):
416+ return apt.Cache()[package_name].versions.get(0).version
417+
418+ def get_click_app_id(self, package_name):
419+ return subprocess.check_output(
420+ ['ubuntu-app-triplet', package_name],
421+ universal_newlines=True).strip()
422+
423+ def get_output_directory(self, app_id):
424+ artifacts_directory = os.environ.get('ADT_ARTIFACTS')
425+ package_directory = '{}-startup-{}'.format(
426+ self.package_name, datetime.datetime.now().strftime(
427+ '%Y%m%d-%H%M%S'))
428+ return os.path.join(
429+ artifacts_directory, 'lttng_traces', package_directory, app_id)
430+
431+ def start_app(self, launch_argument):
432+ subprocess.check_call(
433+ ['ubuntu-app-launch', launch_argument], timeout=15)
434+
435+ def stop_app(self, launch_argument):
436+ subprocess.check_call(['ubuntu-app-stop', launch_argument])
437+
438+ def setUp(self):
439+ super().setUp()
440+ app_id = self.get_app_id(self.package_name)
441+ output_directory = self.get_output_directory(app_id)
442+ self.useFixture(LTTngStarted(self.package_name, output_directory))
443+
444+ def test_app_startup(self):
445+ if self.package_name in debian_packages:
446+ launch_argument = self.package_name
447+ else:
448+ launch_argument = self.get_click_app_id(self.package_name)
449+
450+ self.addCleanup(self.stop_app, launch_argument)
451+ self.start_app(launch_argument)
452+ # XXX it would be better to exit as soon as the trace point is found.
453+ # Ask if that is possible. --elopio - 2014-11-12
454+ time.sleep(30)
455+
456+
457+if __name__ == '__main__':
458+ # We can not just use the default discover because it prints the results
459+ # in stderr.
460+ unittest.main(
461+ testRunner=unittest.TextTestRunner(stream=sys.stdout, verbosity=2))

Subscribers

People subscribed via source and target branches