Fixing the base interface _run_interface function for NiftyRegCommand #2005

byvernault · 2017-05-10T17:21:15Z

Dear Nipype Team,

In our previous release of the niftyreg interfaces, int the base interface, we added some line of code to set at least by default the number of thread use by niftyreg tools.

I just realized this morning that the code I used isn't correct since I am not checking if the key exists in the dictionary. I edited with @effigies advice from the issue #2004 .

Basically, I am setting the self.inputs.environ['OMP_NUM_THREADS'] to the environ variable if it exists, else to the num_threads if not found (default 1).

I ran a node using the interface RegAladin and it worked. I tested as well one interface for NiftySeg and it worked as well.

Let me know if you think I should change something from those few lines.

Kind Regards,

… for setting the OMP default value.

oesteban · 2017-05-10T17:51:14Z

nipype/interfaces/niftyreg/base.py

- if not isdefined(self.inputs.environ['OMP_NUM_THREADS']):
- self.inputs.environ['OMP_NUM_THREADS'] = self.num_threads
+ omp_key = 'OMP_NUM_THREADS'
+ if omp_key in os.environ:


I'd suggest this instead of the if .. else:
self.inputs.environ[omp_key] = os.getenv(omp_key, '%d' % self.num_threads)

chrisgorgo · 2017-05-10T17:55:18Z

This PR is probably not the right place for it, but the OMP_NUM_THREADS variable also controls threads in AFNI commands. We should abstract this away as a separate class (say OpenMPInterface and OpenMPInterfaceInputSpec) and use multi inheritance.

oesteban · 2017-05-10T18:14:55Z

I was coming back to this and, correct me if I'm wrong, but this PR does not actually fix the issue:

If OMP_NUM_THREADS is defined in the environment (e.g. you run export OMP_NUM_THREADS=4 before nipype), then nipype will automatically load that in the interface environment.

So it is not really necessary to make sure of setting it again, it will be already in the environment of the call to subprocess.

Then, I've noticed several issues in the code (sorry I reviewed the PR too quickly before):

self.numthreads vs. self.num_threads (https://github.com/mmodat/nipype/blob/fcae4f71a5f208ad51215c322ca5e6ac3af99fd3/nipype/interfaces/niftyreg/base.py#L117 and https://github.com/mmodat/nipype/blob/fcae4f71a5f208ad51215c322ca5e6ac3af99fd3/nipype/interfaces/niftyreg/base.py#L112). Additionally, since self.num[_]?threads is not set in the __init__, the use of this member is very risky. Please use setattr/getattr.
self.numthreads is not necessary. Why not just:

def _run_interface(self, runtime): if isdefined(self.inputs.omp_core_val): self.inputs.environ['OMP_NUM_THREADS'] = '%d' % self.input.omp_core_val return super(NiftyRegCommand, self)._run_interface(runtime)

Then, the format_arg function would go away.

If OMP_NUM_THREADS is not specified (not defined in the environment and omp_core_val is not defined), then I think nifty* should handle that situation. (meaning, I don't think it is a good idea to define a default value here).

oesteban · 2017-05-10T18:32:16Z

@effigies just reminded me that BaseInterface has a num_threads property (which will be used in MultiProc to see if there are free cpus for an upcoming node). You can set it using self.num_threads safely. Something like:

def _run_interface(self, runtime): if isdefined(self.inputs.omp_core_val): self.inputs.environ['OMP_NUM_THREADS'] = '%d' % self.input.omp_core_val if self.inputs.environ.get('OMP_NUM_THREADS') is not None: self.num_threads = int(self.inputs.environ.get('OMP_NUM_THREADS')) return super(NiftyRegCommand, self)._run_interface(runtime)

satra · 2017-05-10T19:43:10Z

@oesteban - i don't think we should modify self.inputs.environ inside _run_interface. inside that function runtime.environ should be used.

also self.num_threads should be set whenever self.input.omp_core_val is set, which is why i suggested adopting things in afni.

when the multiproc plugin executes, it will want to know how many threads to use, _run_interface is too late to set that value.

essentially here are the scenarios:

nothing set in interface, nothing in environment, everything defaults to 1 thread
nothing in interface, OMP_NUM_THREADS in environment, interface to use OMP_NUM_THREADS
omp_core_val is set, should override runtime.environ. since the nifti tools can invoke this internally, the safest option would be to unset OMP_NUM_THREADS if it exists in environment.

byvernault · 2017-05-10T20:39:41Z

hmm so how do you want me to proceed?

I can also remove the option from the interface and let nipype take care of this and the nifty executables are going to use the OMP_NUM_THREADS value from the env variables.

WDYT?

oesteban · 2017-05-10T20:52:11Z

As @chrisfilo said and @satra confirmed, I think we should give this a general solution.

satra · 2017-05-10T21:04:26Z

@oesteban - i really would like to release 0.13 before the general solution. can you see if the solution below works?

@byvernault - so for these interfaces i would suggest the following, and remove overloading _run_interface:

diff --git a/nipype/interfaces/niftyreg/base.py b/nipype/interfaces/niftyreg/base.py index a7672166f..0c7e22d60 100644 --- a/nipype/interfaces/niftyreg/base.py +++ b/nipype/interfaces/niftyreg/base.py @@ -47,8 +47,8 @@ def no_nifty_package(cmd='reg_f3d'): class NiftyRegCommandInputSpec(CommandLineInputSpec): """Input Spec for niftyreg interfaces.""" # Set the number of omp thread to use - omp_core_val = traits.Int(desc='Number of openmp thread to use', - argstr='-omp %i') + omp_core_val = traits.Int(1, desc='Number of openmp thread to use', + argstr='-omp %i', usedefault=True) class NiftyRegCommand(CommandLine): @@ -73,6 +73,15 @@ class NiftyRegCommand(CommandLine): msg = 'The version of NiftyReg differs from the required' msg += '(%s != %s)' warn(msg % (_version, self.required_version)) + self.num_threads = 1 + self.inputs.on_trait_change(self._omp_update, 'omp_core_val') + + def _omp_update(self): + if self.inputs.omp_core_val: + self.inputs.environ = {'OMP_NUM_THREADS': self.inputs.omp_core_val} + self.num_threads = self.inputs.omp_core_val + else: + self.num_threads = 1

satra · 2017-05-10T21:05:40Z

updated the usedefault keyword.

effigies · 2017-05-10T21:14:09Z

@satra I could be wrong, but I think your solution will always set num_threads = 1 unless it's modified after instantiation (so Interface(omp_core_val=2) would not actually set num_threads to 2).

diff --git a/nipype/interfaces/niftyreg/base.py b/nipype/interfaces/niftyreg/base.py index a7672166f..0c7e22d60 100644 --- a/nipype/interfaces/niftyreg/base.py +++ b/nipype/interfaces/niftyreg/base.py @@ -47,8 +47,8 @@ def no_nifty_package(cmd='reg_f3d'): class NiftyRegCommandInputSpec(CommandLineInputSpec): """Input Spec for niftyreg interfaces.""" # Set the number of omp thread to use - omp_core_val = traits.Int(desc='Number of openmp thread to use', - argstr='-omp %i') + omp_core_val = traits.Int(1, desc='Number of openmp thread to use', + argstr='-omp %i', usedefault=True) class NiftyRegCommand(CommandLine): @@ -73,6 +73,15 @@ class NiftyRegCommand(CommandLine): msg = 'The version of NiftyReg differs from the required' msg += '(%s != %s)' warn(msg % (_version, self.required_version)) + self._omp_update() + self.inputs.on_trait_change(self._omp_update, 'omp_core_val') + + def _omp_update(self): + if self.inputs.omp_core_val: + self.inputs.environ['OMP_NUM_THREADS'] = self.inputs.omp_core_val + self.num_threads = self.inputs.omp_core_val + else: + self.num_threads = 1

Also modified your environment update line, as it currently clobbers any user-set self.inputs.environ.

WDYT? (And what should happen to environ in the else case?)

byvernault · 2017-05-10T21:20:10Z

Sure, I will edit the code for this PR tomorrow morning. All the tests will need to be updated with the default value.

I remember you wanted to do a release so I was trying to fix this so those interfaces are working properly.

Thank you for your help guys.

satra · 2017-05-10T21:36:15Z

@effigies and @byvernault - here is an update. if this passes muster, @byvernault - you should be able to patch this in:

diff --git a/nipype/interfaces/niftyreg/base.py b/nipype/interfaces/niftyreg/base.py index a7672166f..fca2a1b97 100644 --- a/nipype/interfaces/niftyreg/base.py +++ b/nipype/interfaces/niftyreg/base.py @@ -26,7 +26,8 @@ import shutil import subprocess from warnings import warn -from ..base import CommandLine, isdefined, CommandLineInputSpec, traits +from ..base import (CommandLine, isdefined, CommandLineInputSpec, traits, + Undefined) from ...utils.filemanip import split_filename @@ -47,8 +48,8 @@ def no_nifty_package(cmd='reg_f3d'): class NiftyRegCommandInputSpec(CommandLineInputSpec): """Input Spec for niftyreg interfaces.""" # Set the number of omp thread to use - omp_core_val = traits.Int(desc='Number of openmp thread to use', - argstr='-omp %i') + omp_core_val = traits.Int(1, desc='Number of openmp thread to use', + argstr='-omp %i', usedefault=True) class NiftyRegCommand(CommandLine): @@ -58,7 +59,10 @@ class NiftyRegCommand(CommandLine): _suffix = '_nr' _min_version = '1.5.30' + input_spec = NiftyRegCommandInputSpec + def __init__(self, required_version=None, **inputs): + self.num_threads = 1 super(NiftyRegCommand, self).__init__(**inputs) self.required_version = required_version _version = self.get_version() @@ -73,6 +77,26 @@ class NiftyRegCommand(CommandLine): msg = 'The version of NiftyReg differs from the required' msg += '(%s != %s)' warn(msg % (_version, self.required_version)) + self.inputs.on_trait_change(self._omp_update, 'omp_core_val') + self.inputs.on_trait_change(self._environ_update, 'environ') + + def _omp_update(self): + if self.inputs.omp_core_val: + self.inputs.environ['OMP_NUM_THREADS'] = \ + str(self.inputs.omp_core_val) + self.num_threads = self.inputs.omp_core_val + else: + if 'OMP_NUM_THREADS' in self.inputs.environ: + del self.inputs.environ['OMP_NUM_THREADS'] + self.num_threads = 1 + + def _environ_update(self): + if self.inputs.environ: + if 'OMP_NUM_THREADS' in self.inputs.environ: + self.inputs.omp_core_val = \ + int(self.inputs.environ['OMP_NUM_THREADS']) + else: + self.inputs.omp_core_val = Undefined def check_version(self): _version = self.get_version()

satra · 2017-05-10T21:37:38Z

one extra else:

+ def _environ_update(self): + if self.inputs.environ: + if 'OMP_NUM_THREADS' in self.inputs.environ: + self.inputs.omp_core_val = \ + int(self.inputs.environ['OMP_NUM_THREADS']) + else: + self.inputs.omp_core_val = Undefined + else: + self.inputs.omp_core_val = Undefined

satra · 2017-05-10T21:39:47Z

here is the full thing (also added an initial sync with a call to self._omp_update()):

diff --git a/nipype/interfaces/niftyreg/base.py b/nipype/interfaces/niftyreg/base.py index a7672166f..ca0e2e634 100644 --- a/nipype/interfaces/niftyreg/base.py +++ b/nipype/interfaces/niftyreg/base.py @@ -26,7 +26,8 @@ import shutil import subprocess from warnings import warn -from ..base import CommandLine, isdefined, CommandLineInputSpec, traits +from ..base import (CommandLine, isdefined, CommandLineInputSpec, traits, + Undefined) from ...utils.filemanip import split_filename @@ -47,8 +48,8 @@ def no_nifty_package(cmd='reg_f3d'): class NiftyRegCommandInputSpec(CommandLineInputSpec): """Input Spec for niftyreg interfaces.""" # Set the number of omp thread to use - omp_core_val = traits.Int(desc='Number of openmp thread to use', - argstr='-omp %i') + omp_core_val = traits.Int(1, desc='Number of openmp thread to use', + argstr='-omp %i', usedefault=True) class NiftyRegCommand(CommandLine): @@ -58,7 +59,10 @@ class NiftyRegCommand(CommandLine): _suffix = '_nr' _min_version = '1.5.30' + input_spec = NiftyRegCommandInputSpec + def __init__(self, required_version=None, **inputs): + self.num_threads = 1 super(NiftyRegCommand, self).__init__(**inputs) self.required_version = required_version _version = self.get_version() @@ -73,6 +77,29 @@ class NiftyRegCommand(CommandLine): msg = 'The version of NiftyReg differs from the required' msg += '(%s != %s)' warn(msg % (_version, self.required_version)) + self.inputs.on_trait_change(self._omp_update, 'omp_core_val') + self.inputs.on_trait_change(self._environ_update, 'environ') + self._omp_update() + + def _omp_update(self): + if self.inputs.omp_core_val: + self.inputs.environ['OMP_NUM_THREADS'] = \ + str(self.inputs.omp_core_val) + self.num_threads = self.inputs.omp_core_val + else: + if 'OMP_NUM_THREADS' in self.inputs.environ: + del self.inputs.environ['OMP_NUM_THREADS'] + self.num_threads = 1 + + def _environ_update(self): + if self.inputs.environ: + if 'OMP_NUM_THREADS' in self.inputs.environ: + self.inputs.omp_core_val = \ + int(self.inputs.environ['OMP_NUM_THREADS']) + else: + self.inputs.omp_core_val = Undefined + else: + self.inputs.omp_core_val = Undefined def check_version(self): _version = self.get_version()

satra · 2017-05-10T21:41:36Z

this tries to keep omp_core_val in sync with OMP_NUM_THREADS

effigies · 2017-05-11T00:56:30Z

@satra @byvernault I want to preface this by saying that I think that patch is fine as it stands.

Two comments, though.

It appears that if omp_core_val is 0, None or Undefined then nipype will count it as one thread, and niftyreg will get no OMP_NUM_THREADS environment variable. Is it the case that niftyreg will only use one thread, or is this a way of telling nipype not to track that resource? If we want num_threads to track actual usage, why not just set OMP_NUM_THREADS to 1?
I want to be sure we're all on the same page about the intended behavior, because my impression (from admittedly intermittent scanning of the threads) is there's been some amount of talking past each other regarding what the people expect the default behavior with regard to that environment variable to be.

The default behavior in this patch is to set omp_core_val and inputs.environ['OMP_NUM_THREADS'] to 1, regardless of the OMP_NUM_THREADS environment variable passed to nipype. Now, I think this is reasonable and that, if someone wants to use the environment variable, they should do so explicitly as:

x = pe.Node(Interface(), name='x') x.inputs.omp_core_val = int(os.get_env('OMP_NUM_THREADS', 1))

Just want to make sure nobody's surprised.

satra · 2017-05-11T02:38:43Z

@effigies - i set the default for omp_core_val to 1. so unless the user changes something, it will default to 1 independent of what the environment variable is.

soon after this release we should generalize the thread pieces across interfaces and clearly indicate preference order.

byvernault · 2017-05-11T07:20:00Z

My intent was that if the user wants to increase the omp_core_val for this node, he can do it by using the options omp_core_val (it's an option in the niftyseg executables).

If it's not set, then I want the interface to use OMP_NUM_THREADS. If Nothing is specify or set, then we use 1.

As it is right now, it worked when I run the interface. The number of core used by the executables is 1 if OMP_NUM_THREADS not present in env variable, OMP_NUM_THREADS value if set and omp_core_val if the user use the options.

With the changes from @satra, even if OMP_NUM_THREADS is set to 2 and the user doesn't specify omp_core_val, the executable is using 1 since we have the options -omp 1. It's the reason I was avoiding setting the omp_core_val to 1 as default. I will prefer to set it to OMP_NUM_THREADS by default and 1 of not set in the env.

See example below:

I am running this:

import nipype.interfaces.niftyreg as niftyreg test_node = niftyreg.RegAladin() test_node.base_dir = '/Users/byvernault/data/test_reg' test_node.inputs.flo_file = '/Users/byvernault/data/jobsdir/test_vessel2gad/\ vessels2gad_registration/extract_slices/mapflow/_extract_slices0/20151203_1439\ 5801041532ChannelRoutinthoks011a1001_extracted.nii.gz' test_node.inputs.ref_file = '/Users/byvernault/data/jobsdir/test_vessel2gad/\ INPUT/gad_0/NIFTI/20151203_14395801041532ChannelRoutinthoks010a1001.nii.gz' test_node.inputs.nosym_flag = True test_node.inputs.maxit_val = 1 test_node.inputs.ln_val = 1 test_node.inputs.lp_val = 1 print test_node.cmdline result = test_node.run() print result.outputs

byvernault:~$ export OMP_NUM_THREADS=2 byvernault:~$ python /Users/byvernault/home-local/code/git/atom-dev/scripts/nipype_dev/test_nipype.py reg_aladin -aff 20151203_14395801041532ChannelRoutinthoks011a1001_extracted_aff.txt -flo /Users/byvernault/data/jobsdir/test_vessel2gad/vessels2gad_registration/extract_slices/mapflow/_extract_slices0/20151203_14395801041532ChannelRoutinthoks011a1001_extracted.nii.gz -ln 1 -lp 1 -maxit 1 -noSym **-omp 1** -ref /Users/byvernault/data/jobsdir/test_vessel2gad/INPUT/gad_0/NIFTI/20151203_14395801041532ChannelRoutinthoks010a1001.nii.gz -res 20151203_14395801041532ChannelRoutinthoks011a1001_extracted_res.nii.gz 170511-08:47:19,694 interface INFO: stdout 2017-05-11T08:47:19.694473:[reg_aladin] 170511-08:47:19,695 interface INFO: stdout 2017-05-11T08:47:19.694473:[reg_aladin] Command line: 170511-08:47:19,695 interface INFO: stdout 2017-05-11T08:47:19.694473:[reg_aladin] reg_aladin -aff 20151203_14395801041532ChannelRoutinthoks011a1001_extracted_aff.txt -flo /Users/byvernault/data/jobsdir/test_vessel2gad/vessels2gad_registration/extract_slices/mapflow/_extract_slices0/20151203_14395801041532ChannelRoutinthoks011a1001_extracted.nii.gz -ln 1 -lp 1 -maxit 1 -noSym -omp 1 -ref /Users/byvernault/data/jobsdir/test_vessel2gad/INPUT/gad_0/NIFTI/20151203_14395801041532ChannelRoutinthoks010a1001.nii.gz -res 20151203_14395801041532ChannelRoutinthoks011a1001_extracted_res.nii.gz 170511-08:47:19,695 interface INFO: stdout 2017-05-11T08:47:19.694473:[reg_aladin] 170511-08:47:19,695 interface INFO: stdout 2017-05-11T08:47:19.694473:[reg_aladin] OpenMP is used with 1 thread(s)

Where we have OMP_NUM_THREADS at 2 but reg_aladin used only 1 threads since -omp 1 was set.

byvernault · 2017-05-11T07:57:12Z

What I can do is all the changes from @satra but set the default value of omp_core_val like below:

omp_core_val = traits.Int(int(os.environ.get('OMP_NUM_THREADS', '1')), desc='Number of openmp thread to use', argstr='-omp %i', usedefault=True)

WDYT?

…core_val present

codecov-io · 2017-05-11T10:36:55Z

Codecov Report

Merging #2005 into master will decrease coverage by 0.02%.
The diff coverage is 42.42%.

@@ Coverage Diff @@ ## master #2005 +/- ## ========================================== - Coverage 72.19% 72.17% -0.03%  ========================================== Files 1132 1130 -2 Lines 56988 56963 -25 Branches 8159 8156 -3 ========================================== - Hits 41144 41112 -32  - Misses 14560 14567 +7  Partials 1284 1284

Flag	Coverage Δ
#smoketests	`72.17% <42.42%> (-0.03%)`	⬇️
#unittests	`69.76% <42.42%> (-0.03%)`	⬇️

Impacted Files	Coverage Δ
...rfaces/niftyreg/tests/test_auto_NiftyRegCommand.py	`100% <ø> (ø)`	⬆️
.../interfaces/niftyreg/tests/test_auto_RegAverage.py	`85.71% <ø> (ø)`	⬆️
...pype/interfaces/niftyreg/tests/test_auto_RegF3D.py	`85.71% <ø> (ø)`	⬆️
...nterfaces/niftyreg/tests/test_auto_RegTransform.py	`85.71% <ø> (ø)`	⬆️
...interfaces/niftyreg/tests/test_auto_RegJacobian.py	`85.71% <ø> (ø)`	⬆️
...pe/interfaces/niftyreg/tests/test_auto_RegTools.py	`85.71% <ø> (ø)`	⬆️
.../interfaces/niftyreg/tests/test_auto_RegMeasure.py	`85.71% <ø> (ø)`	⬆️
...e/interfaces/niftyreg/tests/test_auto_RegAladin.py	`85.71% <ø> (ø)`	⬆️
...interfaces/niftyreg/tests/test_auto_RegResample.py	`85.71% <ø> (ø)`	⬆️
nipype/interfaces/niftyreg/tests/test_regutils.py	`4.46% <0%> (-0.11%)`	⬇️
... and 6 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update e605dfe...4757bec. Read the comment docs.

byvernault · 2017-05-11T10:37:01Z

I added the changes from @satra with my last message for the default value. Is that ok? I can edit again if needed.

I also changed the NiftySegCommand to be a children class of NiftyFitCommand instead of NiftyRegCommand since there are no omp_core_val.

Cheers,

Fixing the base interface _run_interface function for NiftyRegCommand…

fcae4f7

… for setting the OMP default value.

oesteban reviewed May 10, 2017

View reviewed changes

byvernault added 2 commits May 11, 2017 11:34

Adding changes for omp_val_core and editing the tests.

bda7c32

Changing NiftySegCommand to use NiftyFitCommand instead since no omp_…

4757bec

…core_val present

satra merged commit ec86b74 into nipy:master May 11, 2017

effigies mentioned this pull request May 11, 2017

niftyreg interfaces failing because of base and OMP_NUM_THREADS #2004

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Fixing the base interface _run_interface function for NiftyRegCommand #2005

Fixing the base interface _run_interface function for NiftyRegCommand #2005

Uh oh!

byvernault commented May 10, 2017

oesteban May 10, 2017 •

edited

Loading

chrisgorgo commented May 10, 2017

oesteban commented May 10, 2017

oesteban commented May 10, 2017

satra commented May 10, 2017

byvernault commented May 10, 2017

oesteban commented May 10, 2017

satra commented May 10, 2017 •

edited

Loading

satra commented May 10, 2017

effigies commented May 10, 2017

byvernault commented May 10, 2017 •

edited

Loading

satra commented May 10, 2017

satra commented May 10, 2017

satra commented May 10, 2017

satra commented May 10, 2017

effigies commented May 11, 2017

satra commented May 11, 2017

byvernault commented May 11, 2017 •

edited

Loading

byvernault commented May 11, 2017

codecov-io commented May 11, 2017 •

edited

Loading

byvernault commented May 11, 2017

Labels

6 participants

Fixing the base interface _run_interface function for NiftyRegCommand #2005

Fixing the base interface _run_interface function for NiftyRegCommand #2005

Uh oh!

Conversation

byvernault commented May 10, 2017

oesteban May 10, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

chrisgorgo commented May 10, 2017

oesteban commented May 10, 2017

oesteban commented May 10, 2017

satra commented May 10, 2017

byvernault commented May 10, 2017

oesteban commented May 10, 2017

satra commented May 10, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

satra commented May 10, 2017

effigies commented May 10, 2017

byvernault commented May 10, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

satra commented May 10, 2017

satra commented May 10, 2017

satra commented May 10, 2017

satra commented May 10, 2017

effigies commented May 11, 2017

satra commented May 11, 2017

byvernault commented May 11, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

byvernault commented May 11, 2017

codecov-io commented May 11, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

byvernault commented May 11, 2017

Labels

6 participants

oesteban May 10, 2017 •

edited

Loading

satra commented May 10, 2017 •

edited

Loading

byvernault commented May 10, 2017 •

edited

Loading

byvernault commented May 11, 2017 •

edited

Loading

codecov-io commented May 11, 2017 •

edited

Loading