I discovered a slight issue with the Cymru whois expert bot with some strange AS names. For instance, AS266522.

$ dig +short TXT AS266522.asn.cymru.com "266522 | BR | lacnic | 2017-01-20 | Realnet Telecomunicacoes e Multimidia ltda\226\128\143 , BR" 

\226\128\143, in unicode, decodes to u+200f, the right-to-left mark. When the IMQ bot attempts to decode it with result['as_name'] = items[4].encode('latin1').decode('utf8'), it runs into this issue:
UnicodeEncodeError: 'latin-1' codec can't encode character '\u200f' in position 42: ordinal not in range(256)

A solution which just permits decoding special symbols like RTL marks might not be the best, though just discarding these events into the dump file is also not a good option.

Various online tools for finding public data on ASes, found online via a quick search, give varying results.

• ipinfo.io displays it in 3 different ways ("injected" RTL, â€ and a�)
• qrator displays it as &#8207
• bigdatacloud displays it as 8207
• bgpview displays it as ��
• etc.

I suppose there's no existing expectation of invalid UTF-8 sequences or special symbols like RTL marks in AS names, nor any "standard" way of handling them, and I feel iffy about passing arbitrary symbols like that on to other systems.

Are there any ideas for how to handle cases like this? The simplest option would be to simply pass the undecoded string on in case the decoding step fails.