CLIAdvancedArrow String Representation

How CloudQuery represents some Arrow types as strings

Some types in Arrow are highly specialized and often there isn’t a one to one mapping to a database type. For example, Arrow has a Date32 type which is a 32-bit integer representing the number of days since the UNIX epoch. This type is not supported by most databases, so CloudQuery has to represent it as a string.

Here is list of all Arrow types and their string representation.

Arrow TypeString RepresentationExample
Binary, FixedSizeBinary, LargeBinarybase64 StdEncodingYQ==
Booleanstrconv.FormatBool output”true”
Decimal128, Decimal256GetOneForMarshal.(string)“12345”
List, FixedSizeListstring(GetOneForMarshal.(json.RawMessage))[ 1, 2, 3 ]
MonthIntervalint32 as string”123”
DayTimeIntervalJSON {“days”:int32, “milliseconds”:int32}{"days":1, "milliseconds":234}
MonthDayNanoIntervalJSON {“months”:int32, “days”:int32, “nanoseconds”: int64}{"months":1, "days":2, "nanoseconds": 34567}
StructJSON{"key": ["values", "value2"]}
String, LargeStringAs is”foo”
Uint8, Uint16, Uint32, Uint64strconv.FormatUint”123”
Int8, Int16, Int32, Int64strconv.FormatInt”-123”
Float16GetOneForMarshal.(string)“123.45”
Float32, Float64strconv.FormatFloat”123.45”
TimestampYYYY-MM-DD HH:mm:ss.SSSSSSSSS, no timezone info”2006-01-02 15:04:05.999999999”
Time32HH:mm:ss or HH:mm:ss.SSS, depending on precision”15:04:05.000”
Time64HH:mm:ss.SSSSSS or HH:mm:ss.SSSSSSSSS, depending on precision”15:04:05.000000”
Date32, Date64YYYY-MM-DD”2006-01-02”
DurationNumeric amount and time unit, concatenated”12345ms”
SparseUnion, DenseUnionGetOneForMarshal.(string)

Null values (in nested types) are represented as (null)

CloudQuery Extension Types

Extension TypeUnderlying Arrow TypeString RepresentationExample
UUIDFixedSizeBinary(16)Formatted UUID”123e4567-e89b-12d3-a456-426614174000”
MAC AddressBinaryFormatted MAC Address”01:23:45:67:89:ab”
InetStringAs is”192.168.1.0/24”
JSONStringAs is{"key": "value"}
Using an Offline LicenseUsing CloudQuery Docker Registry Integration Inside a Containerized Environment