feat: export NextToken to check token info

[xujianhai666]

• easy for custom iter.ReadMapCB, we need to use NextToken to check is there more token, for example, we check ',' token to know the end.

Change-Id: Ia335036825df098503f7008bf013b9442fe14298

[codecov]

Codecov Report

Merging #442 into master will increase coverage by 0.04%.
The diff coverage is 0%.

@@ Coverage Diff @@ ## master #442 +/- ## ========================================== + Coverage 86.46% 86.51% +0.04%  ========================================== Files 41 41 Lines 5106 5108 +2 ========================================== + Hits 4415 4419 +4  + Misses 555 553 -2  Partials 136 136

Impacted Files	Coverage Δ
iter.go	89.25% <0%> (-0.85%)	⬇️
reflect_struct_decoder.go	81.89% <0%> (+0.53%)	⬆️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 49c900e...ae60458. Read the comment docs.

[xujianhai666]

@AllenX2018

[AllenX2018]

1. I'm sorry but I don't exactly understand why you need this function? Could you show some code about the use case for detailed description?
2. Please add test for your commit.

[xujianhai666]

1. I'm sorry but I don't exactly understand why you need this function? Could you show some code about the use case for detailed description?
2. Please add test for your commit.
   @AllenX2018

1. fastjson of alibaba is special format, I should compatible with that map format, which there is no quota between "{" or "}", more information refer by "https://github.com/apache/rocketmq-client-go/pull/481/files#diff-01a0b164a18584e31bcfdbeaba02929cR262"
2. func NextToken has been for long time, so ut could be ommited

[xujianhai666]

ping @AllenX2018

[ShannonDing]

hello, @AllenX2018 ,
we strongly hope to have this interface to parse the map from fastjson, Could you plz help to review it?

[AllenX2018]

@ShannonDing @xujianhai666 I'm sorry for the late reply and I've been busy these days. I will check it this weekend.

[AllenX2018]

I've checkout your reference information here. I'm sorry that I'm not familiar with alibaba fastjson. But I guess what you want to decode is a json like:

{ "xxx":{ {"yyy":"xxx"}:{"zzz","xxx"}, {"yyy":"xxx"}:{"zzz","xxx"}, } } 

If so, I suggest that you register an extension to decode such a "struct" type map key. You can check the example below:

func main() { data := []byte(` { "mqTable":{ {"whatEverKey":"key1"}:{"whatEverValue":"value1"}, {"whatEverKey":"key2"}:{"whatEverValue":"value2"} } }`) table := make(map[keyStruct]valueStruct) jsoniter.ConfigCompatibleWithStandardLibrary.RegisterExtension(&structKeyMapExt{}) iter := jsoniter.ParseBytes(jsoniter.ConfigCompatibleWithStandardLibrary, data) _ = iter.ReadMapCB(func(iterator *jsoniter.Iterator, key string) bool { if key == "mqTable" { iter.ReadVal(&table)	} return true	}) if iter.Error != nil && iter.Error != io.EOF { fmt.Println(iter.Error)	} else { fmt.Printf("%+v\n", table)	} } type keyStruct struct { WhatEverKey string `json:"whatEverKey"` } type valueStruct struct { WhatEverValue string `json:"whatEverValue"` } type structKeyMapExt struct {	jsoniter.DummyExtension } func (e *structKeyMapExt) CreateMapKeyDecoder(typ reflect2.Type) jsoniter.ValDecoder { if typ == reflect2.TypeOf(keyStruct{}) { return &structKeyDecoder{}	} return nil } type structKeyDecoder struct { } func (dec *structKeyDecoder) Decode(ptr unsafe.Pointer, iter *jsoniter.Iterator) { iter.ReadVal((*keyStruct)(ptr)) }

You can try it to decode your json. But please use the json-iterator/go with the latest commit on master branch(don't use v1.1.9 because it has a bug). Please let me know if you have any question.

[alvarolm]

@AllenX2018

I support this commit in the case of custom Unmarshal implementation, since its required to check for bytes left after unmarshaling
https://github.com/json-iterator/go/blob/master/config.go#L349

why a custom Unmarshal implementation could be needed ?
since codecs only provide unsafe.Pointer and *Iterator but not the direct value of v that may be needed to avoid reflection cost or just direct handling

thanks !

[alvarolm]

just realized that this commit will not compile, since only exports the definition without updating references, just created #481