[FEATURE REQUEST]: Expose metadata

[thrixton]

This is more a question than a feature request.

When parsing JSON files, I need to sanitize the field names so field with spaces becomes field_with_spaces.
I want to preserve the original name as well, metadata about the column if you like :)

There is a metadata field on StructField, but it is internal.
Why is this internal, is it possible or desirable to expose it?

[imback82]

Thanks @thrixton brining this up.

It is internal for now since it is not properly mapped to the Metadata class on the Scala side: https://github.com/apache/spark/blob/master/sql/catalyst/src/main/scala/org/apache/spark/sql/types/Metadata.scala. If we want to expose this as public, we need to properly wrap the field.

[thrixton]

Good to know @imback82.
Currently, I'm covered as I just modify the JSON DOM directly and then parse it using StructType and have the internals visible so can access Metadata.
I guess at some point full support will go into the backlog.

[imback82]

Let me keep this open so we can follow up on this feature.