ENH: add StringMethods (.str accessor) to Index, fixes #9068

[mortada]

fixes #9068

[mortada]

@jreback please let me know what you think of this, thanks

[jreback]

this looks good
pls add a release note in v0.16.1

[jreback]

may also want to update the docs to mention u can do this on Index (I think it's in text.rst iirc)

[mortada]

@jreback both release note and text.rst have been updated

[shoyer]

Can you add a check here to exclude string methods on MultiIndex objects, too? They have object dtype, too.

[mortada]

@shoyer you mean a simple check like this?

 if isinstance(self, MultiIndex): raise AttributeError(".str accessor is not supported for MultiIndex objects") 

[shoyer]

Yes, that looks about right

[shoyer]

Normally, I would suggest only putting this accessor on the appropriate types directly, but it's a little tricky with MultiIndex because it's a subclass of the generic Index type (which is what string indexes area)

[jreback]

I think better to check the inferred type
which will be string or mixed for s multi index (iirc)

[mortada]

@shoyer @jreback I improved the checking here according to your recommendations - in the case of Series it's the same logic as before and in the case of Index it uses the inferred_type attribute

[sinhrks]

Nice, but 2 points:

• There are some function returns bool, and Index(dtype=bool) is not usable very much (related to API: Index should support __inverse__ ops #8875). It may be better to return np.array in these cases.
• Should prohibit to use return_type="frame" in split ?

[jreback]

@sinhrks can you elaborate on the first point (about bool)? an example?

@mortada

I think add a tests for all of the .str methods (e.g. to test the return types issue), I think you will have to raise if return_type='frame' is defined

[mortada]

@jreback just added a check and unit test for the case of Index and return_type='frame'

@sinhrks I'm looking into the bool issue you referenced

[shoyer]

@jreback @mortada some str methods return booleans, e.g., str.isalpha. For series, it makes sense to return a As @sinhrks points out, we probably don't want to return a new Index object for these values, but rather a plain numpy array.

[mortada]

@shoyer, @sinhrks I see, indeed returning np.array in the boolean case seems a lot more useful.

@jreback an example would be:

idx = Index(['a1', 'a2', 'b1', 'b2']) s = Series(range(4), index=idx) 

If we return an np.array then an expression like

s[s.index.str.startswith('a')] 

would work naturally, whereas if we return a boolean Index that's not very useful

[mortada]

Just added a check for the boolean Index case, it will now return np.array instead. Please take a look, thanks!

[shoyer]

It would be nice to add an example or two here, both to show off the new feature and to illustrate the behavior with boolean output.

[mortada]

good idea, will do

[jreback]

use is_bool_dtype(result)

[mortada]

good catch I'll change this

[jreback]

@mortada yeh I think we need to do something for

Index(['foo_bar','bah_baz']).str.split('_',return_type='frame')

[sinhrks]

@mortada Thanks for quick action:)

[jorisvandenbossche]

Didn't look in detail into this, but a quick question: what is the return type? I suppose in general a new index? But what if the method returns multiple elements for each entry? (eg returning a frame for a series)?

[mortada]

@jorisvandenbossche the return type is Index unless the dtype is bool, in which case the return type will be np.array.

if the method returns multiple elements it will be an Index of lists, and won't be a DataFrame. If the user specifies return_type='frame' it will raise. Here's an example:

In [1]: from pandas import Index In [2]: idx = Index(['a b c', 'd e', 'f']) In [3]: idx.str.split() Out[3]: Index([['a', 'b', 'c'], ['d', 'e'], ['f']], dtype='object') In [4]: idx.str.split(return_type='frame') --------------------------------------------------------------------------- ValueError Traceback (most recent call last) <ipython-input-4-d22b631e0f0f> in <module>() ----> 1 idx.str.split(return_type='frame') /Users/mortada_mehyar/code/github/pandas/pandas/core/strings.py in split(self, pat, n, return_type) 987 @copy(str_split) 988 def split(self, pat=None, n=-1, return_type='series'): --> 989 result = str_split(self.series, pat, n=n, return_type=return_type) 990 return self._wrap_result(result) 991 /Users/mortada_mehyar/code/github/pandas/pandas/core/strings.py in str_split(arr, pat, n, return_type) 652 raise ValueError("return_type must be {'series', 'frame'}") 653 if return_type == 'frame' and isinstance(arr, Index): --> 654 raise ValueError("return_type='frame' is not supported for string " 655 "methods on Index") 656 if pat is None: ValueError: return_type='frame' is not supported for string methods on Index 

[mortada]

@shoyer added examples to doc
@jreback now using is_bool_dtype

please take a look, thanks.

[jorisvandenbossche]

By the way, I think this is a very nice feature, especially useful for cleaning up your column names (stripping whitespace at beginning or end -> typical problem you see on SO, replacing whitespace with _, convert to all lower case, ..) For the docs, maybe it would be useful to also focus on this and give an example where the column names are used (but maybe not needed for this PR)

[jreback]

@jorisvandenbossche I don't think its a problem if certain methods which have a return_type='frame' raise if you try to call them on an Index. The only other option is to actually return a frame as I think the purpose is what you mentioned above, to allow cleaning type ops on an Index, which you expect to get an Index.

Another option is to exclude the methods entirely from the Index (meaning ones that have a default return_type='frame')

[mortada]

@jreback I added reture_type='index' as a alias for 'series' as you suggested, please take a look

[jreback]

lgtm. @jorisvandenbossche

[jorisvandenbossche]

@mortada Are you sure you committed it? I still only see the frame/series options (or I am overlooking it completely ..)

[mortada]

@jorisvandenbossche oops I had to do another rebase and missed the last commit, thanks for catching!

[mortada]

@jorisvandenbossche it's fixed, please take a look thanks

[jorisvandenbossche]

Maybe make this {'series'/'index', 'frame'} to make it more clear they are aliases?

[jorisvandenbossche]

The thing I am thinking now: won't it be confusing for users to get a Index back even if he/she supplies return_type='series' on a Index.str.split, or to get a series back with Series.str.split(.., return_type='index')?

I was also thinking of another option: an expand keyword (or another name), that indicates for False: give same dimension back (so for series/index keep it a series/index), and for True: expand series to dataframe.
This would then be a duplicate for return_type of course. But the return_type was only introduced in 0.15.1 (and for partition it is still in a PR), so if we want to change this: better now than later. Or has this ship sailed?

But maybe we shouldn't block this PR with this discussion and open a new issue for this. So I am OK with merging.

[jreback]

@jorisvandenbossche

how about we just do

return_type='same'|'expand' to satisfy this need? (and can be easily back-compat)

but let's do that in another issue / PR

[mortada]

@jreback @jorisvandenbossche I'm happy to do another PR for the return_type

[jreback]

ok, merging this, but @mortada I would appreciate another issue (and PR!) for the return_type issue.

[jorisvandenbossche]

@mortada really a nice improvement!

I will open an new issue to discuss the return_type thing

[mortada]

@jorisvandenbossche thanks!

Another thing you had mentioned was expanding the docs to cover some questions on SO about cleaning up column names. Could you please point me to some of those SO questions? I can add more examples to the docs.

[jorisvandenbossche]

@mortada I don't have a specific SO question on my mind, but you often see questions like: "why does df['my_col'] give a KeyError as it is in the columns?" with response -> "check df.columns, maybe you have starting or trailing spaces in the names" (something you don't easily see in the console output, you can only detect it through the alignment)

So for me, this are typical ways to clean up column names:

• remove starting/trailing whitespace (eg leftover of reading it in)
• convert all columns names to lower case (eg something typical you would want for writing it to certain databases/on certain platforms to not have to deal with case sensitivity issues)
• remove alle whitespace in the column names, or convert all whitespaces to underscores ("my col" -> "my_col")

[shoyer]

@mortada very nice, thanks!