FIX: Fixes for nodal/global/fiber stats outputs of connectivity mapping #348

swederik · 2012-03-29T00:32:03Z

Numerous fixes for the CSV outputs of the connectivity mapping workflows (foremost, Function interfaces are now incorporated)
Tractography filtered by intersections is no longer removed as an unnecessary output
Structural tessellation workflow incorporated into advanced diffusion MRI tutorial, added into resulting CFF file
Added exists=True tags to Parcellation inputs/outputs

Update: Just tested the group mrtrix tutorial on a new machine and it seems to work perfectly. Ready to merge unless someone thinks otherwise.

chrisgorgo · 2012-03-30T11:14:37Z

examples/dmri_group_connectivity_mrtrix.py

whoops, was easier to test with one group... fixed now

chrisgorgo · 2012-04-04T13:55:20Z

I get a bunch of failing nodes while running dmri_group_connectivity_mrtrix. Here are the crashlogs http://dl.dropbox.com/u/412959/dmri_group_connectivity_mrtrix_errors.zip

swederik · 2012-04-04T13:59:55Z

Hmm, those all look related to the Freesurfer directory for one of the subjects....
Could you try downloading the data package again? I know at one point of the subjects was missing the files in the /surf folder, so maybe you grabbed it before I fixed that.

Is there a way to open the .npz files?

chrisgorgo · 2012-04-04T14:01:45Z

How did you figure it out without opening the npz files?

use "nipype_display_crash" to view those logs

swederik · 2012-04-04T14:09:05Z

Heh, I had the same error logs about a month ago... Plus all those interfaces convert surfaces or volumes from .mgz at some point.

chrisgorgo · 2012-04-04T17:27:13Z

I downloaded the newest dataset. I run out of memory! And I have 16G!

filo@filolaptop:/mnt/data/nipype_examples$ nipype_display_crash /mnt/data/nipype_examples/crash-20120404-181729-filo-csdeconv.a0.npz 120404-18:21:41,162 workflow DEBUG: networkx 1.4 dev or higher detected File: /mnt/data/nipype_examples/crash-20120404-181729-filo-csdeconv.a0.npz Node: l1pipeline.connectivity.mapping.csdeconv.a0 Working directory: /mnt/data/nipype_examples/dmri_group_connectivity_mrtrix/l1pipeline/connectivity/mapping/_subject_id_pat10/csdeconv Node inputs: args = <undefined> directions_file = <undefined> encoding_file = <undefined> environ = {'DISPLAY': ':1'} filter_file = <undefined> ignore_exception = False in_file = <undefined> iterations = <undefined> lambda_value = <undefined> mask_image = <undefined> maximum_harmonic_order = 6.0 normalise = <undefined> out_filename = <undefined> response_file = <undefined> threshold_value = <undefined> Traceback: Traceback (most recent call last): File "/usr/local/lib/python2.7/dist-packages/nipype-0.0.0-py2.7.egg/nipype/pipeline/plugins/multiproc.py", line 15, in run_node result['result'] = node.run(updatehash=updatehash) File "/usr/local/lib/python2.7/dist-packages/nipype-0.0.0-py2.7.egg/nipype/pipeline/engine.py", line 1126, in run self._run_interface() File "/usr/local/lib/python2.7/dist-packages/nipype-0.0.0-py2.7.egg/nipype/pipeline/engine.py", line 1221, in _run_interface self._result = self._run_command(execute) File "/usr/local/lib/python2.7/dist-packages/nipype-0.0.0-py2.7.egg/nipype/pipeline/engine.py", line 1345, in _run_command result = self._interface.run() File "/usr/local/lib/python2.7/dist-packages/nipype-0.0.0-py2.7.egg/nipype/interfaces/base.py", line 820, in run runtime = self._run_interface(runtime) File "/usr/local/lib/python2.7/dist-packages/nipype-0.0.0-py2.7.egg/nipype/interfaces/base.py", line 1096, in _run_interface self.raise_exception(runtime) File "/usr/local/lib/python2.7/dist-packages/nipype-0.0.0-py2.7.egg/nipype/interfaces/base.py", line 1061, in raise_exception raise RuntimeError(message) RuntimeError: Command: csdeconv -grad /mnt/data/nipype_examples/dmri_group_connectivity_mrtrix/l1pipeline/connectivity/mapping/_subject_id_pat10/fsl2mrtrix/bvecs_bvals.txt -mask /mnt/data/nipype_examples/dmri_group_connectivity_mrtrix/l1pipeline/connectivity/mapping/_subject_id_pat10/gen_WM_mask/vol0000_flirt_merged_WMProb.mif -lmax 6.0 /mnt/data/nipype_examples/dmri_group_connectivity_mrtrix/l1pipeline/connectivity/mapping/eddycorrect/_subject_id_pat10/merge/vol0000_flirt_merged.nii /mnt/data/nipype_examples/dmri_group_connectivity_mrtrix/l1pipeline/connectivity/mapping/_subject_id_pat10/estimateresponse/vol0000_flirt_merged_ER.mif vol0000_flirt_merged_CSD.mif Standard output: Standard error: terminate called after throwing an instance of 'std::bad_alloc' what(): std::bad_alloc Aborted Return code: 134 Interface ConstrainedSphericalDeconvolution failed to run. Inputs: args = <undefined> directions_file = <undefined> encoding_file = /mnt/data/nipype_examples/dmri_group_connectivity_mrtrix/l1pipeline/connectivity/mapping/_subject_id_pat10/fsl2mrtrix/bvecs_bvals.txt environ = {'DISPLAY': ':1'} filter_file = <undefined> ignore_exception = False in_file = /mnt/data/nipype_examples/dmri_group_connectivity_mrtrix/l1pipeline/connectivity/mapping/eddycorrect/_subject_id_pat10/merge/vol0000_flirt_merged.nii iterations = <undefined> lambda_value = <undefined> mask_image = /mnt/data/nipype_examples/dmri_group_connectivity_mrtrix/l1pipeline/connectivity/mapping/_subject_id_pat10/gen_WM_mask/vol0000_flirt_merged_WMProb.mif maximum_harmonic_order = 6.0 normalise = <undefined> out_filename = <undefined> response_file = /mnt/data/nipype_examples/dmri_group_connectivity_mrtrix/l1pipeline/connectivity/mapping/_subject_id_pat10/estimateresponse/vol0000_flirt_merged_ER.mif threshold_value = <undefined>

swederik · 2012-04-04T17:39:54Z

Haha wow very strange...
I've had that before but it somehow went away... Which version of mrtrix are you running? The latest is like 2.9.10 or something, and it fixed a number of problems I had (off the top of my head, Nifti conversion in mrconvert).

Definitely shouldn't be running out of memory, mine doesn't even come close and I've run those pipelines on machines with only 4gb.

Was it being run with multiproc?

swederik · 2012-04-05T12:21:52Z

Wow, strangely enough I'm getting the same memory error on my end now...

chrisgorgo · 2012-04-05T14:05:52Z

I've tried both 0.2.9 and 0.2.10 with the same result.

chrisgorgo · 2012-04-15T13:57:42Z

Maybe we should send a bug report upstream?

swederik · 2012-04-23T14:53:53Z

Okay, so I've narrowed this problem down to the Estimate Response node. It appears that the subjects in which csdeconv crashes all seem to have spherical response profiles.

http://dl.dropbox.com/u/315714/response_profile.png

No clue how to fix it yet though....

swederik · 2012-04-23T15:49:10Z

Turns out it's related to the absolute thresholding on the FA image to obtain the single-fibre voxels.. works fine if it's set back to the default (0.7).

swederik · 2012-04-23T16:33:05Z

nipype/interfaces/cmtk/parcellation.py

i don't know why this looks like the spacing is wrong but it's fine in my editor....

…st exist

…t file

chrisgorgo · 2012-05-06T13:50:34Z

Is this still WIP?

I have noticed that Parcellate node iterates over labels and calls freesurfer tools internally using a CommandLine class. Have you considered wrapping those tools and turning Parcellate into a workfow with MapNodes?

…t file

…st exist

…t file

satra · 2012-08-07T03:06:21Z

where are we with this?

swederik · 2012-08-07T06:44:40Z

Not sure. I'm convinced it's ready to merge but chris said that the
NetworkXMetrics node was rerunning. I've been unable to replicate the bug
on two Ubuntu machines and one Mac... so I'm still convinced.

It's pretty annoying that the tutorial is so damn big and long though.
Takes forever to test.
On Aug 7, 2012 5:06 AM, "Satrajit Ghosh" notifications@github.com wrote:

where are we with this?

—
Reply to this email directly or view it on GitHubhttps://github.com//pull/348#issuecomment-7542669.

swederik · 2012-10-28T15:42:12Z

Hey guys,

I'm beginning to wonder if we will ever close this PR.
I've tried to reproduce Chris' bug on two ubuntu machines and a mac and I don't seem to get any rerunning.

Is there any other reason not to merge it?

chrisgorgo · 2012-11-08T18:01:40Z

I'll ry it on my new machine. We will merge it! Yes we can!

On 28 October 2012 16:42, Erik Ziegler notifications@github.com wrote:

Hey guys,

I'm beginning to wonder if we will ever close this PR.
I've tried to reproduce Chris' bug on two ubuntu machines and a mac and I
don't seem to get any rerunning.

Is there any other reason not to merge it?

—
Reply to this email directly or view it on GitHubhttps://github.com//pull/348#issuecomment-9846857.

swederik · 2012-11-18T23:34:06Z

8 months later! We finally did it!

I'll start cleaning up some other branches for new PRs :)

chrisgorgo · 2012-11-19T08:53:08Z

Mind that I had to make some changes - please update your master.

On 19 November 2012 00:34, Erik Ziegler notifications@github.com wrote:

8 months later! We finally did it!

I'll start cleaning up some other branches for new PRs :)

—
Reply to this email directly or view it on GitHubhttps://github.com//pull/348#issuecomment-10497032.

chrisgorgo reviewed Mar 30, 2012
View reviewed changes

swederik reviewed Apr 23, 2012
View reviewed changes

nipype/interfaces/cmtk/parcellation.py

Copy link

Member Author

swederik Apr 23, 2012

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i don't know why this looks like the spacing is wrong but it's fine in my editor....

Erik Ziegler added 13 commits May 2, 2012 22:39

Fix for single subject global network csvs'

bef2b1e

Changed CSV concatenation to use Function interfaces

ea7039b

ConcatCSV interface and CreateMatrix output fixes

d9b8337

Fix for creatematrix outputs

68cd441

Freesurfer directory from environment variable

57c03f6

Added os import

1cdf09e

Fix for std::badalloc, lmax float to int, parcellation subject dir mu…

97a9201

…st exist

Uncommented subject lists

e3afdb5

Spacing fix

c7c3a1f

Fixed os.path.exists call

58853df

Group mrtrix tutorial runs properly

6827935

Typo fix

05009e8

Number/percentage of tracts are now output from CreateMatrix as a .ma…

d5cd625

…t file

Erik Ziegler added 23 commits May 30, 2012 18:12

Uncommented subject lists

073f8fe

Spacing fix

acf772b

Fixed os.path.exists call

96a982c

Group mrtrix tutorial runs properly

7c8b5bb

Typo fix

defcebd

Number/percentage of tracts are now output from CreateMatrix as a .ma…

73969f9

…t file

Improved CSV merging. Single subj and group level fiber csvs

97118b9

Fix for single subject global network csvs'

cd1d717

Changed CSV concatenation to use Function interfaces

9577862

ConcatCSV interface and CreateMatrix output fixes

43a7ef4

Fix for creatematrix outputs

152ce7c

Fix for std::badalloc, lmax float to int, parcellation subject dir mu…

0163bec

…st exist

Uncommented subject lists

66a8431

Spacing fix

d0aadb9

Fixed os.path.exists call

dfa7513

Group mrtrix tutorial runs properly

7f50678

Typo fix

b5199a8

Number/percentage of tracts are now output from CreateMatrix as a .ma…

2a2fae9

…t file

Fixes for global CSV merging

c898b35

Added more global metrics

b64965f

Merge branch 'enh/fibercsv' into fix/csv

9156745

CSVFibers fix

5a9c02a

Fixed an unintended comment

e730994

chrisgorgo merged commit e730994 into nipy:master Nov 18, 2012

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

FIX: Fixes for nodal/global/fiber stats outputs of connectivity mapping #348

FIX: Fixes for nodal/global/fiber stats outputs of connectivity mapping #348

Uh oh!

swederik commented Mar 29, 2012

chrisgorgo Mar 30, 2012

swederik Apr 3, 2012

chrisgorgo commented Apr 4, 2012

swederik commented Apr 4, 2012

chrisgorgo commented Apr 4, 2012

swederik commented Apr 4, 2012

chrisgorgo commented Apr 4, 2012

swederik commented Apr 4, 2012

swederik commented Apr 5, 2012

chrisgorgo commented Apr 5, 2012

chrisgorgo commented Apr 15, 2012

swederik commented Apr 23, 2012

swederik commented Apr 23, 2012

swederik Apr 23, 2012

chrisgorgo commented May 6, 2012

satra commented Aug 7, 2012

swederik commented Aug 7, 2012

swederik commented Oct 28, 2012

chrisgorgo commented Nov 8, 2012

swederik commented Nov 18, 2012

chrisgorgo commented Nov 19, 2012

Labels

3 participants

FIX: Fixes for nodal/global/fiber stats outputs of connectivity mapping #348

FIX: Fixes for nodal/global/fiber stats outputs of connectivity mapping #348

Uh oh!

Conversation

swederik commented Mar 29, 2012

chrisgorgo Mar 30, 2012

Choose a reason for hiding this comment

swederik Apr 3, 2012

Choose a reason for hiding this comment

chrisgorgo commented Apr 4, 2012

swederik commented Apr 4, 2012

chrisgorgo commented Apr 4, 2012

swederik commented Apr 4, 2012

chrisgorgo commented Apr 4, 2012

swederik commented Apr 4, 2012

swederik commented Apr 5, 2012

chrisgorgo commented Apr 5, 2012

chrisgorgo commented Apr 15, 2012

swederik commented Apr 23, 2012

swederik commented Apr 23, 2012

swederik Apr 23, 2012

Choose a reason for hiding this comment

chrisgorgo commented May 6, 2012

satra commented Aug 7, 2012

swederik commented Aug 7, 2012

swederik commented Oct 28, 2012

chrisgorgo commented Nov 8, 2012

swederik commented Nov 18, 2012

chrisgorgo commented Nov 19, 2012

Labels

3 participants