Skip to content
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
77 commits
Select commit Hold shift + click to select a range
cb53be7
ENH add max_level and ignore_keys configuration to nested_to_records
bhavaniravi Nov 22, 2018
0972746
ENH extend max_level and ignore keys to
bhavaniravi Nov 22, 2018
5a5c708
fix pep8 issues
bhavaniravi Nov 22, 2018
be7ec0e
add whatsnew to doc string
bhavaniravi Nov 22, 2018
a79e126
add testcase with large max_level
bhavaniravi Nov 23, 2018
cd12a23
add explation for flatten if condition
bhavaniravi Nov 23, 2018
d3b3503
update doc_string and built documentation
bhavaniravi Nov 23, 2018
4ec60bc
fix json normalize records path issue
bhavaniravi Nov 27, 2018
e001264
Merge branch 'master' into enhanced_json_normalize
bhavaniravi Nov 27, 2018
5c88339
Merge branch 'master' of git://github.com/pandas-dev/pandas into json…
bhavaniravi Dec 30, 2018
55f7b1c
fix merge conflict
bhavaniravi Jan 3, 2019
1af2bfc
fix testcase error
bhavaniravi Jan 3, 2019
882a2ca
add nested flattening to json_normalize
bhavaniravi Jan 3, 2019
caba6db
fixed pep8 issues
bhavaniravi Jan 3, 2019
4e22c69
fix merge conflict
bhavaniravi Jan 3, 2019
c2eff85
fix issues with doc string
bhavaniravi Jan 4, 2019
247124f
modify test case to paramaetized
bhavaniravi Jan 4, 2019
ab15869
fix issues with pep8
bhavaniravi Jan 10, 2019
26bf967
fix pep8 build fail
bhavaniravi Jan 16, 2019
fca2a27
fix testcase failure, inconsistent column order
bhavaniravi Feb 5, 2019
7a58456
fix documentation issues
bhavaniravi Mar 19, 2019
f3d25e3
fix merge conflicts with upstream
bhavaniravi Mar 19, 2019
7a1297d
Merge branch 'master' of git://github.com/pandas-dev/pandas into enha…
bhavaniravi Apr 20, 2019
177c750
fix testcase failure np.nan converted into str on line 328
bhavaniravi Apr 20, 2019
cb82bca
remove get_pip file
bhavaniravi Apr 20, 2019
2a7b966
rename test func test_max_level_with_record_prefix
bhavaniravi Apr 20, 2019
4635591
fix pep8 over-intended line
bhavaniravi Apr 21, 2019
22fd84e
fix docstring formatting issues
bhavaniravi Apr 21, 2019
2e407e3
convert to a fixture
bhavaniravi Apr 21, 2019
cf27cae
convert to inline data
bhavaniravi Apr 21, 2019
124fbd9
fix docstring formatting issues
bhavaniravi Apr 21, 2019
7b65999
fix docstring formatting issues
bhavaniravi Apr 21, 2019
03d3d23
add github issue id to test case
bhavaniravi Apr 22, 2019
8e61a04
fix pep8 flake issues
bhavaniravi Apr 22, 2019
b808d5a
Merge branch 'master' of git://github.com/pandas-dev/pandas into enha…
bhavaniravi Apr 22, 2019
0eaea30
Merge branch 'master' of git://github.com/pandas-dev/pandas into enha…
bhavaniravi Apr 23, 2019
837ba18
Merge branch 'master' of git://github.com/pandas-dev/pandas into enha…
bhavaniravi Apr 26, 2019
217d4ae
Merge branch 'master' of git://github.com/pandas-dev/pandas into enha…
bhavaniravi Apr 30, 2019
b2fc133
seperate ignore keys and max level implementation
bhavaniravi Jun 16, 2019
acf1137
Merge branch 'master' of git://github.com/pandas-dev/pandas into enha…
bhavaniravi Jun 16, 2019
ff30152
fix errors on validating doc strings
bhavaniravi Jun 16, 2019
fa2ecee
fix flake8 issues on test cases
bhavaniravi Jun 16, 2019
aed2db5
remove unwanted file
bhavaniravi Jun 16, 2019
f5dacd6
move fixture to the calling function
bhavaniravi Jun 17, 2019
33e2504
fix pep8 issue
bhavaniravi Jun 17, 2019
699d696
type annotated methods
bhavaniravi Jun 17, 2019
a91f27a
update docs based on review comments
bhavaniravi Jun 17, 2019
0a04cdb
fix flake8 issue E252
bhavaniravi Jun 19, 2019
e4e586d
add issue id to testcases
bhavaniravi Jun 19, 2019
62a35db
fix syntax issue with respect to typing in 3.5
bhavaniravi Jun 19, 2019
d113401
fix docstring and typing issue
bhavaniravi Jun 20, 2019
bfa62cf
fix typing linting issue
bhavaniravi Jun 20, 2019
53b6bcb
Merge branch 'master' of git://github.com/pandas-dev/pandas into enha…
bhavaniravi Jun 20, 2019
2bc829b
updated typing from defaults to typing specific structures
bhavaniravi Jun 23, 2019
d6a7cc7
merged max_level testcases into a parametrized tests
bhavaniravi Jun 23, 2019
a69ad2b
Merge branch 'master' of git://github.com/pandas-dev/pandas into enha…
bhavaniravi Jun 23, 2019
b0133d2
fix documentation issue
bhavaniravi Jun 23, 2019
1564d49
updated typing from defaults to typing specific structures
bhavaniravi Jun 23, 2019
463adc7
move expected data as parameters of test
bhavaniravi Jun 23, 2019
bbf894a
Merge branch 'master' of git://github.com/pandas-dev/pandas into enha…
bhavaniravi Jun 23, 2019
775472e
checks fix and parameteize test case
bhavaniravi Jun 25, 2019
f8f550a
fix pep8 spacing issue
bhavaniravi Jun 25, 2019
4ddf0cc
resolve merge conflict
bhavaniravi Jun 25, 2019
9f2d356
fix import sort issue
bhavaniravi Jun 25, 2019
f3ff665
fix type checking and documentation error
bhavaniravi Jun 26, 2019
20432ad
Merge branch 'master' of git://github.com/pandas-dev/pandas into enha…
bhavaniravi Jun 26, 2019
311b898
parametrize none test case, generalized typing
bhavaniravi Jun 27, 2019
7850db7
Merge branch 'master' of git://github.com/pandas-dev/pandas into enha…
bhavaniravi Jun 27, 2019
676c7f1
remove test.json
bhavaniravi Jun 27, 2019
e2796d4
split large_max_level test case into a seperate function
bhavaniravi Jun 28, 2019
69a0d43
add doc string
bhavaniravi Jun 28, 2019
3a80a4d
add docs to io.rst and whatsnew
bhavaniravi Jun 28, 2019
c288c2e
add missing import
bhavaniravi Jun 28, 2019
f96d8fb
fix typing issue and docs update
bhavaniravi Jun 29, 2019
3ec85bf
fix typing check issues
bhavaniravi Jun 29, 2019
ba1d983
liniting issues fix in documentation
bhavaniravi Jun 29, 2019
4b754a0
fix liniting issue
bhavaniravi Jun 29, 2019
File filter

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
checks fix and parameteize test case
  • Loading branch information
bhavaniravi committed Jun 25, 2019
commit 775472ece8a4568fabb8c24d38ab8cd8b69133a3
2 changes: 1 addition & 1 deletion pandas/io/json/normalize.py
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
# JSON normalization routines

from collections import defaultdict
from typing import Union, List, Dict, DefaultDict
from typing import DefaultDict, Dict, List, Union
import copy

import numpy as np
Expand Down
67 changes: 27 additions & 40 deletions pandas/tests/io/json/test_normalize.py
Original file line number Diff line number Diff line change
Expand Up @@ -168,8 +168,6 @@ def test_more_deeply_nested(self, deep_nested):

result = json_normalize(deep_nested, ['states', 'cities'],
meta=['country', ['states', 'name']])
# meta_prefix={'states': 'state_'})

ex_data = {'country': ['USA'] * 4 + ['Germany'] * 3,
'states.name': ['California', 'California', 'Ohio', 'Ohio',
'Bayern', 'Nordrhein-Westfalen',
Expand Down Expand Up @@ -294,9 +292,28 @@ def test_missing_field(self, author_missing_data):
expected = DataFrame(ex_data)
tm.assert_frame_equal(result, expected)

@pytest.mark.parametrize("max_level", [0, 1])
def test_max_level_with_records_path(self,
max_level):
@pytest.mark.parametrize("max_level,expected", [
(0, [{"TextField": "Some text",
'UserField': {'Id': 'ID001',
'Name': 'Name001'},
"CreatedBy": {"Name": "User001"},
'Image': {'a': 'b'}},
{"TextField": "Some text",
'UserField': {'Id': 'ID001',
'Name': 'Name001'},
"CreatedBy": {"Name": "User001"},
'Image': {'a': 'b'}}]),
(1, [{"TextField": "Some text",
"UserField.Id": "ID001",
"UserField.Name": "Name001",
"CreatedBy": {"Name": "User001"},
'Image': {'a': 'b'}},
{"TextField": "Some text",
"UserField.Id": "ID001",
"UserField.Name": "Name001",
"CreatedBy": {"Name": "User001"},
'Image': {'a': 'b'}}])])
def test_max_level_with_records_path(self, max_level, expected):
# GH23843: Enhanced JSON normalize
test_input = [{'CreatedBy': {'Name': 'User001'},
'Lookup': [{'TextField': 'Some text',
Expand All @@ -311,32 +328,11 @@ def test_max_level_with_records_path(self,
{'foo': 'something2', 'bar': 'else2'}]
}]

expected_data = {0: [{"TextField": "Some text",
'UserField': {'Id': 'ID001',
'Name': 'Name001'},
"CreatedBy": {"Name": "User001"},
'Image': {'a': 'b'}},
{"TextField": "Some text",
'UserField': {'Id': 'ID001',
'Name': 'Name001'},
"CreatedBy": {"Name": "User001"},
'Image': {'a': 'b'}}],
1: [{"TextField": "Some text",
"UserField.Id": "ID001",
"UserField.Name": "Name001",
"CreatedBy": {"Name": "User001"},
'Image': {'a': 'b'}},
{"TextField": "Some text",
"UserField.Id": "ID001",
"UserField.Name": "Name001",
"CreatedBy": {"Name": "User001"},
'Image': {'a': 'b'}}]}
expected_data = expected_data[max_level]
result = json_normalize(test_input,
record_path=["Lookup"],
meta=[["CreatedBy"], ["Image"]],
max_level=max_level)
expected_df = DataFrame(data=expected_data,
expected_df = DataFrame(data=expected,
columns=result.columns.values)
tm.assert_equal(expected_df, result)

Expand Down Expand Up @@ -401,19 +397,10 @@ def test_missing_meta(self, missing_metadata):
record_path='addresses',
meta='name',
errors='ignore')
ex_data = [
{'city': 'Massillon',
'number': 9562,
'state': 'OH',
'street': 'Morris St.',
'zip': 44646,
'name': 'Alice'},
{'city': 'Elizabethton',
'number': 8449,
'state': 'TN',
'street': 'Spring St.',
'zip': 37643,
'name': np.nan}
ex_data =[
['Massillon', 9562, 'OH', 'Morris St.', 44646, 'Alice'],
['Elizabethton', 8449, 'TN', 'Spring St.', 37643, np.nan]

]
columns = ['city', 'number', 'state', 'street', 'zip', 'name']
expected = DataFrame(ex_data, columns=columns)
Expand Down