Juju Charms Collection
postgresql package

Merge lp:~stub/charms/precise/postgresql/cleanups into lp:charms/postgresql

  1. Precise Pangolin (12.04)
  2. cleanups
  3. Merge into trunk
Proposed by Stuart Bishop
Status: Merged
Approved by: Mark Mims
Approved revision: 72
Merged at revision: 62
Proposed branch: lp:~stub/charms/precise/postgresql/cleanups
Merge into: lp:charms/postgresql
Prerequisite: lp:~stub/charms/precise/postgresql/bug-1205286
Diff against target: 569 lines (+127/-183)
7 files modified
README.md (+37/-40)
charm-helpers.yaml (+1/-1)
hooks/charmhelpers/core/host.py (+7/-5)
hooks/hooks.py (+49/-126)
metadata.yaml (+12/-8)
templates/start_conf.tmpl (+13/-0)
test.py (+8/-3)
To merge this branch: bzr merge lp:~stub/charms/precise/postgresql/cleanups
Reviewer Review Type Date Requested Status
Mark Mims (community) Approve
Review via email: mp+181738@code.launchpad.net

Description of the change

Update documentation and code cleanups.

To post a comment you must log in.
lp:~stub/charms/precise/postgresql/cleanups updated
72. By Stuart Bishop

Add missing template

Revision history for this message
Mark Mims (mark-mims) :
review: Approve

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
1=== modified file 'README.md'
2--- README.md 2013-08-07 13:50:02 +0000
3+++ README.md 2013-08-23 09:40:09 +0000
4@@ -41,15 +41,6 @@
5 maintains replication for you, using standard PostgreSQL streaming
6 replication.
7
8- - Multiple services linked using 'master'/'slave' relationships. A
9- single service can be the 'master', and multiple services connected
10- to this master in a 'slave' role. Each service can contain multiple
11- units; the 'master' service will contain a single 'master' unit and
12- remaining units all 'hot standby'. The 'slave' services will only
13- contain 'hot standby' units. 'Cascading replication is not
14- supported', so do not attempt to relate an existing 'slave' service
15- as a 'master' to another service.
16-
17
18 To setup a single 'standalone' service::
19
20@@ -61,41 +52,34 @@
21
22 juju add-unit pg-a
23
24-To deploy a new service containing a 'master' and a 'hot standby'::
25-
26- juju deploy -n 2 postgresql pg-b
27-
28-
29-To relate a PostgreSQL service as a 'slave' of another PostgreSQL service.
30-**Caution** - this destroys the existing databases in the pg-b service::
31-
32- juju add-relation pg-a:master pg-b:slave
33-
34-
35-To setup a client using a PostgreSQL database, in this case OpenERP and
36-its web front end. Note that OpenERP requires an administrative level
37-connection::
38+To deploy a new service containing a 'master' and two 'hot standbys'::
39+
40+ juju deploy -n 3 postgresql pg-b
41+
42+You can remove units as normal. If the master unit is removed, failover
43+occurs and the most up to date 'hot standby' is promoted to 'master'.
44+The 'db-relation-changed' and 'db-admin-relation-changed' hooks are
45+fired, letting clients adjust::
46+
47+ juju remove-unit pg-b/0
48+
49+
50+To setup a client using a PostgreSQL database, in this case a vanilla
51+Django installation listening on port 8080::
52
53 juju deploy postgresql
54- juju deploy postgresql pg-standby
55- juju deploy openerp-web
56- juju deploy openerp-server
57-
58- juju add-relation postgresql:master pg-standby:slave
59- juju add-relation openerp-server:db postgresql:db-admin
60- juju add-relation openerp-web openerp-server
61-
62- juju expose openerp-web
63- juju expose openerp-server
64+ juju deploy python-django
65+ juju deploy gunicorn
66+ juju add-relation python-django postgresql:db
67+ juju add-relation python-django gunicorn
68+ juju expose python-django
69
70
71 ## Restrictions
72
73 - Do not attempt to relate client charms to a PostgreSQL service
74 containing multiple units unless you know the charm supports
75- a replicated service. You can use a 'master'/'slave' relationship
76- to create a redundant copy of your database until the client charms
77- are updated.
78+ a replicated service.
79
80 - You cannot host multiple units in a single juju container. This is
81 problematic as some PostgreSQL features, such as tablespaces, use
82@@ -103,10 +87,23 @@
83
84 # Interacting with the Postgresql Service
85
86-Typically, you just need to join a the `db` relation, and a user and database
87-will be created for you. For more advanced uses, you can join the `db-admin`
88-relation, and a super user will be created. Using this account, you can
89-manipulate all other aspects of the database.
90+At a minimum, you just need to join a the `db` relation, and a user and
91+database will be created for you. For more complex environments,
92+you can provide the `database` name allowing multiple services to share
93+the same database. A client may also wish to defer its setup until the
94+unit name is listed in `allowed-units`, to avoid attempting to connect
95+to a database before it has been authorized.
96+
97+The `db-admin` relation may be used similarly to the `db` relation.
98+The automatically generated user for `db-admin` relations is a
99+PostgreSQL superuser.
100+
101+## During db-relation-joined
102+
103+### the client service provides:
104+
105+- `database`: Optional. The name of the database to use. The postgresql
106+ service will create it if necessary.
107
108 ## During db-relation-changed
109
110
111=== modified file 'charm-helpers.yaml'
112--- charm-helpers.yaml 2013-08-23 09:40:09 +0000
113+++ charm-helpers.yaml 2013-08-23 09:40:09 +0000
114@@ -1,4 +1,4 @@
115 destination: hooks/charmhelpers
116-branch: lp:charm-helpers
117+branch: lp:~stub/charm-helpers/bug-1214793-service-wrappers
118 include:
119 - core
120
121=== modified file 'hooks/charmhelpers/core/host.py'
122--- hooks/charmhelpers/core/host.py 2013-08-23 09:40:09 +0000
123+++ hooks/charmhelpers/core/host.py 2013-08-23 09:40:09 +0000
124@@ -20,20 +20,22 @@
125
126
127 def service_start(service_name):
128- service('start', service_name)
129+ return service('start', service_name)
130
131
132 def service_stop(service_name):
133- service('stop', service_name)
134+ return service('stop', service_name)
135
136
137 def service_restart(service_name):
138- service('restart', service_name)
139+ return service('restart', service_name)
140
141
142 def service_reload(service_name, restart_on_failure=False):
143- if not service('reload', service_name) and restart_on_failure:
144- service('restart', service_name)
145+ service_result = service('reload', service_name)
146+ if not service_result and restart_on_failure:
147+ service_result = service('restart', service_name)
148+ return service_result
149
150
151 def service(action, service_name):
152
153=== modified file 'hooks/hooks.py'
154--- hooks/hooks.py 2013-08-23 09:40:09 +0000
155+++ hooks/hooks.py 2013-08-23 09:40:09 +0000
156@@ -32,20 +32,13 @@
157 return Template(*args, **kw)
158
159
160-def write_file(path, contents, owner='root', group='root', perms=0o444):
161- '''Temporary alternative to charm-helpers write_file().
162-
163- charm-helpers' write_file() magic makes it useless for any file
164- containing curly brackets, so work around for now until the feature
165- can be discussed.
166- '''
167- log("Writing file {} {}:{} {:o}".format(path, owner, group, perms))
168- uid = getpwnam(owner).pw_uid
169- gid = getgrnam(group).gr_gid
170- dest_fd = os.open(path, os.O_WRONLY | os.O_TRUNC | os.O_CREAT, perms)
171- os.fchown(dest_fd, uid, gid)
172- with os.fdopen(dest_fd, 'w') as destfile:
173- destfile.write(str(contents))
174+def log(msg, lvl=INFO):
175+ # Per Bug #1208787, log messages sent via juju-log are being lost.
176+ # Spit messages out to a log file to work around the problem.
177+ myname = hookenv.local_unit().replace('/', '-')
178+ with open('/tmp/{}-debug.log'.format(myname), 'a') as f:
179+ f.write('{}: {}\n'.format(lvl, msg))
180+ hookenv.log(msg, lvl)
181
182
183 class State(dict):
184@@ -87,8 +80,6 @@
185
186 replication_state = dict(client_state)
187
188- add(replication_state, 'public_ssh_key')
189- add(replication_state, 'ssh_host_key')
190 add(replication_state, 'replication_password')
191 add(replication_state, 'wal_received_offset')
192 add(replication_state, 'following')
193@@ -189,30 +180,18 @@
194 return output
195
196
197-#------------------------------------------------------------------------------
198-# Enable/disable service start by manipulating policy-rc.d
199-#------------------------------------------------------------------------------
200-def enable_service_start(service):
201- ### NOTE: doesn't implement per-service, this can be an issue
202- ### for colocated charms (subordinates)
203- log("enabling {} start by policy-rc.d".format(service))
204- if os.path.exists('/usr/sbin/policy-rc.d'):
205- os.unlink('/usr/sbin/policy-rc.d')
206- return True
207- return False
208-
209-
210-def disable_service_start(service):
211- log("disabling {} start by policy-rc.d".format(service))
212- policy_rc = '/usr/sbin/policy-rc.d'
213- policy_rc_tmp = "{}.tmp".format(policy_rc)
214- open(policy_rc_tmp, 'w').write("""#!/bin/bash
215-[[ "$1"-"$2" == %s-start ]] && exit 101
216-exit 0
217-EOF
218-""" % service)
219- os.chmod(policy_rc_tmp, 0755)
220- os.rename(policy_rc_tmp, policy_rc)
221+def postgresql_autostart(enabled):
222+ if enabled:
223+ log("Enabling PostgreSQL startup in {}".format(startup_file))
224+ mode = 'auto'
225+ else:
226+ log("Disabling PostgreSQL startup in {}".format(startup_file))
227+ mode = 'manual'
228+ startup_file = os.path.join(postgresql_config_dir, 'start.conf')
229+ contents = Template(open("templates/start_conf.tmpl").read()).render(
230+ {'mode': mode})
231+ host.write_file(
232+ startup_file, contents, 'postgres', 'postgres', perms=0o644)
233
234
235 def run(command, exit_on_error=True):
236@@ -229,10 +208,6 @@
237 raise
238
239
240-#------------------------------------------------------------------------------
241-# postgresql_stop, postgresql_start, postgresql_is_running:
242-# wrappers over invoke-rc.d, with extra check for postgresql_is_running()
243-#------------------------------------------------------------------------------
244 def postgresql_is_running():
245 # init script always return true (9.1), add extra check to make it useful
246 status, output = commands.getstatusoutput("invoke-rc.d postgresql status")
247@@ -244,17 +219,12 @@
248
249
250 def postgresql_stop():
251- status, output = commands.getstatusoutput("invoke-rc.d postgresql stop")
252- if status != 0:
253- return False
254+ host.service_stop('postgresql')
255 return not postgresql_is_running()
256
257
258 def postgresql_start():
259- status, output = commands.getstatusoutput("invoke-rc.d postgresql start")
260- if status != 0:
261- log(output, CRITICAL)
262- return False
263+ host.service_start('postgresql')
264 return postgresql_is_running()
265
266
267@@ -275,12 +245,9 @@
268 last_warning = time.time()
269 time.sleep(5)
270
271- status, output = \
272- commands.getstatusoutput("invoke-rc.d postgresql restart")
273- if status != 0:
274- return False
275+ return host.service_restart('postgresql')
276 else:
277- postgresql_start()
278+ return host.service_start('postgresql')
279
280 # Store a copy of our known live configuration so
281 # postgresql_reload_or_restart() can make good choices.
282@@ -406,7 +373,7 @@
283 # Return it as pg_config
284 pg_config = Template(
285 open("templates/postgresql.conf.tmpl").read()).render(config_data)
286- write_file(
287+ host.write_file(
288 postgresql_config, pg_config,
289 owner="postgres", group="postgres", perms=0600)
290
291@@ -419,7 +386,7 @@
292 ident_data = {}
293 pg_ident_template = Template(
294 open("templates/pg_ident.conf.tmpl").read())
295- write_file(
296+ host.write_file(
297 postgresql_ident, pg_ident_template.render(ident_data),
298 owner="postgres", group="postgres", perms=0600)
299
300@@ -520,7 +487,7 @@
301 relation_data.append(local_replication)
302
303 pg_hba_template = Template(open("templates/pg_hba.conf.tmpl").read())
304- write_file(
305+ host.write_file(
306 postgresql_hba, pg_hba_template.render(access_list=relation_data),
307 owner="postgres", group="postgres", perms=0600)
308 postgresql_reload()
309@@ -542,7 +509,7 @@
310 }
311 crontab_template = Template(
312 open("templates/postgres.cron.tmpl").read()).render(crontab_data)
313- write_file('/etc/cron.d/postgres', crontab_template, perms=0600)
314+ host.write_file('/etc/cron.d/postgres', crontab_template, perms=0600)
315
316
317 def create_recovery_conf(master_host, restart_on_change=False):
318@@ -557,7 +524,7 @@
319 'host': master_host,
320 'password': local_state['replication_password']})
321 log(recovery_conf, DEBUG)
322- write_file(
323+ host.write_file(
324 os.path.join(postgresql_cluster_dir, 'recovery.conf'),
325 recovery_conf, owner="postgres", group="postgres", perms=0o600)
326
327@@ -782,7 +749,7 @@
328 volid = volume_get_volume_id()
329 if not volid:
330 ## Invalid configuration (whether ephemeral, or permanent)
331- disable_service_start("postgresql")
332+ postgresql_autostart(False)
333 postgresql_stop()
334 mounts = volume_get_all_mounted()
335 if mounts:
336@@ -797,9 +764,9 @@
337 ## config_changed_volume_apply will stop the service if it founds
338 ## it necessary, ie: new volume setup
339 if config_changed_volume_apply():
340- enable_service_start("postgresql")
341+ postgresql_autostart(True)
342 else:
343- disable_service_start("postgresql")
344+ postgresql_autostart(False)
345 postgresql_stop()
346 mounts = volume_get_all_mounted()
347 if mounts:
348@@ -863,10 +830,10 @@
349 open("templates/dump-pg-db.tmpl").read()).render(paths)
350 backup_job = Template(
351 open("templates/pg_backup_job.tmpl").read()).render(paths)
352- write_file(
353+ host.write_file(
354 '{}/dump-pg-db'.format(postgresql_scripts_dir),
355 dump_script, perms=0755)
356- write_file(
357+ host.write_file(
358 '{}/pg_backup_job'.format(postgresql_scripts_dir),
359 backup_job, perms=0755)
360 install_postgresql_crontab(postgresql_crontab)
361@@ -1256,56 +1223,6 @@
362 local_state.save()
363
364
365-def ensure_local_ssh():
366- """Generate SSH keys for postgres user.
367-
368- The public key is stored in public_ssh_key on the relation.
369-
370- Bidirectional SSH access is required by repmgr.
371- """
372- comment = 'repmgr key for {}'.format(os.environ['JUJU_UNIT_NAME'])
373- if not os.path.isdir(postgres_ssh_dir):
374- host.mkdir(postgres_ssh_dir, "postgres", "postgres", 0o700)
375- if not os.path.exists(postgres_ssh_private_key):
376- run("sudo -u postgres -H ssh-keygen -q -t rsa -C '{}' -N '' "
377- "-f '{}'".format(comment, postgres_ssh_private_key))
378- public_key = open(postgres_ssh_public_key, 'r').read().strip()
379- host_key = open('/etc/ssh/ssh_host_ecdsa_key.pub').read().strip()
380- local_state['public_ssh_key'] = public_key
381- local_state['ssh_host_key'] = host_key
382- local_state.publish()
383-
384-
385-def authorize_remote_ssh():
386- """Generate the SSH authorized_keys file."""
387- authorized_units = set()
388- authorized_keys = set()
389- known_hosts = set()
390- for relid in hookenv.relation_ids('replication'):
391- for unit in hookenv.related_units(relid):
392- relation = hookenv.relation_get(unit=unit, rid=relid)
393- public_key = relation.get('public_ssh_key', None)
394- if public_key:
395- authorized_units.add(unit)
396- authorized_keys.add(public_key)
397- known_hosts.add('{} {}'.format(
398- relation['private-address'], relation['ssh_host_key']))
399-
400- # Generate known_hosts
401- write_file(
402- postgres_ssh_known_hosts, '\n'.join(known_hosts),
403- owner="postgres", group="postgres", perms=0o644)
404-
405- # Generate authorized_keys
406- write_file(
407- postgres_ssh_authorized_keys, '\n'.join(authorized_keys),
408- owner="postgres", group="postgres", perms=0o400)
409-
410- # Publish details, so relation knows they have been granted access.
411- local_state['authorized'] = authorized_units
412- local_state.publish()
413-
414-
415 @contextmanager
416 def pgpass():
417 passwords = {}
418@@ -1665,6 +1582,16 @@
419 config_changed()
420
421
422+@contextmanager
423+def switch_cwd(new_working_directory):
424+ org_dir = os.getcwd()
425+ os.chdir(new_working_directory)
426+ try:
427+ yield new_working_directory
428+ finally:
429+ os.chdir(org_dir)
430+
431+
432 def clone_database(master_unit, master_host):
433 with pgpass():
434 postgresql_stop()
435@@ -1680,7 +1607,10 @@
436 shutil.rmtree(postgresql_cluster_dir)
437
438 try:
439- output = subprocess.check_output(cmd)
440+ # Change directory the postgres user can read.
441+ with switch_cwd('/tmp'):
442+ # Run the sudo command.
443+ output = subprocess.check_output(cmd, stderr=subprocess.STDOUT)
444 log(output, DEBUG)
445 # Debian by default expects SSL certificates in the datadir.
446 os.symlink(
447@@ -1690,7 +1620,7 @@
448 '/etc/ssl/private/ssl-cert-snakeoil.key',
449 os.path.join(postgresql_cluster_dir, 'server.key'))
450 create_recovery_conf(master_host)
451- except subprocess.CalledProcessError, x:
452+ except subprocess.CalledProcessError as x:
453 # We failed, and this cluster is broken. Rebuild a
454 # working cluster so start/stop etc. works and we
455 # can retry hooks again. Even assuming the charm is
456@@ -1817,7 +1747,7 @@
457 check_file_age -w {} -c {} -f {}".format(warn_age, crit_age, backup_log))
458
459 if os.path.isfile('/etc/init.d/nagios-nrpe-server'):
460- subprocess.call(['service', 'nagios-nrpe-server', 'reload'])
461+ host.service_reload('nagios-nrpe-server')
462
463
464 ###############################################################################
465@@ -1841,12 +1771,6 @@
466 config_data['backup_dir'].strip() or
467 os.path.join(postgresql_data_dir, 'backups'))
468 postgresql_logs_dir = os.path.join(postgresql_data_dir, 'logs')
469-postgres_ssh_dir = os.path.expanduser('~postgres/.ssh')
470-postgres_ssh_public_key = os.path.join(postgres_ssh_dir, 'id_rsa.pub')
471-postgres_ssh_private_key = os.path.join(postgres_ssh_dir, 'id_rsa')
472-postgres_ssh_authorized_keys = os.path.join(postgres_ssh_dir,
473- 'authorized_keys')
474-postgres_ssh_known_hosts = os.path.join(postgres_ssh_dir, 'known_hosts')
475 hook_name = os.path.basename(sys.argv[0])
476 replication_relation_types = ['master', 'slave', 'replication']
477 local_state = State('local_state.pickle')
478@@ -1859,5 +1783,4 @@
479 if hookenv.relation_id():
480 log("Relation {} with {}".format(
481 hookenv.relation_id(), hookenv.remote_unit()))
482-
483 hooks.execute(sys.argv)
484
485=== modified file 'metadata.yaml'
486--- metadata.yaml 2013-06-25 11:29:13 +0000
487+++ metadata.yaml 2013-08-23 09:40:09 +0000
488@@ -1,13 +1,17 @@
489 name: postgresql
490-summary: "object-relational SQL database (supported version)"
491+summary: "PostgreSQL object-relational SQL database (supported version)"
492 description: |
493- PostgreSQL is a fully featured object-relational database management
494- system. It supports a large part of the SQL standard and is designed
495- to be extensible by users in many aspects. Some of the features are:
496- ACID transactions, foreign keys, views, sequences, subqueries,
497- triggers, user-defined types and functions, outer joins, multiversion
498- concurrency control. Graphical user interfaces and bindings for many
499- programming languages are available as well.
500+ PostgreSQL is a powerful, open source object-relational database system.
501+ It has more than 15 years of active development and a proven
502+ architecture that has earned it a strong reputation for reliability,
503+ data integrity, and correctness. It is fully ACID compliant, has full
504+ support for foreign keys, joins, views, triggers, and stored procedures
505+ (in multiple languages). It includes most SQL:2008 data types, including
506+ INTEGER, NUMERIC, BOOLEAN, CHAR, VARCHAR, DATE, INTERVAL, and TIMESTAMP.
507+ It also supports storage of binary large objects, including pictures,
508+ sounds, or video. It has native programming interfaces for C/C++, Java,
509+ .Net, Perl, Python, Ruby, Tcl, ODBC, among others, and exceptional
510+ documentation (http://www.postgresql.org/docs/manuals/).
511 maintainer: Stuart Bishop <stuart.bishop@canonical.com>
512 categories:
513 - databases
514
515=== added file 'templates/start_conf.tmpl'
516--- templates/start_conf.tmpl 1970-01-01 00:00:00 +0000
517+++ templates/start_conf.tmpl 2013-08-23 09:40:09 +0000
518@@ -0,0 +1,13 @@
519+#
520+# This file is managed by Juju.
521+#
522+# Automatic startup configuration
523+# auto: automatically start/stop the cluster in the init script
524+# manual: do not start/stop in init scripts, but allow manual startup with
525+# pg_ctlcluster
526+# disabled: do not allow manual startup with pg_ctlcluster (this can be easily
527+# circumvented and is only meant to be a small protection for
528+# accidents).
529+
530+{{mode}}
531+
532
533=== modified file 'test.py'
534--- test.py 2013-08-23 09:40:09 +0000
535+++ test.py 2013-08-23 09:40:09 +0000
536@@ -128,13 +128,19 @@
537 # enough that our system is probably stable. This means we have
538 # extremely slow and flaky tests, but that is possibly better
539 # than no tests.
540- time.sleep(30)
541+ time.sleep(45)
542
543 def setUp(self):
544 DEBUG("JujuFixture.setUp()")
545 super(JujuFixture, self).setUp()
546 self.reset()
547- self.addCleanup(self.reset)
548+ # Optionally, don't teardown services and machines after running
549+ # a test. If a subsequent test is run, they will be torn down at
550+ # that point. This option is only useful when running a single
551+ # test, or when the test harness is set to abort after the first
552+ # failed test.
553+ if not os.environ.get('TEST_DONT_TEARDOWN_JUJU', False):
554+ self.addCleanup(self.reset)
555
556 def reset(self):
557 DEBUG("JujuFixture.reset()")
558@@ -286,7 +292,6 @@
559 result = self.sql('SELECT TRUE', dbname='postgres')
560 self.assertEqual(result, [['t']])
561
562-
563 def is_master(self, postgres_unit, dbname=None):
564 is_master = self.sql(
565 'SELECT NOT pg_is_in_recovery()',
566
567=== added directory 'tests'
568=== added symlink 'tests/01_pg_testsuite.test'
569=== target is u'../test.py'

Subscribers

People subscribed via source and target branches