Ruby

[ruby/prism] Fix incorrect paring when using invalid regexp options

Fixes https://github.com/ruby/prism/pull/2617.

There was an issue with the lexer as follows.
The following are valid regexp options:

$ bundle exec ruby -Ilib -rprism -ve 'p Prism.lex("/x/io").value.map {|token| token[0].type }' ruby 3.3.0 (2023-12-25 revision https://github.com/ruby/prism/commit/5124f9ac75) [x86_64-darwin22] [:REGEXP_BEGIN, :STRING_CONTENT, :REGEXP_END, :EOF] 

The following are invalid regexp options. Unnecessary the IDENTIFIER token is appearing:

$ bundle exec ruby -Ilib -rprism -ve 'p Prism.lex("/x/az").value.map {|token| token[0].type }' ruby 3.3.0 (2023-12-25 revision https://github.com/ruby/prism/commit/5124f9ac75) [x86_64-darwin22] [:REGEXP_BEGIN, :STRING_CONTENT, :REGEXP_END, :IDENTIFIER, :EOF] 

As a behavior of Ruby, when given A to Z and a to z, they act as invalid regexp options. e.g.,

$ ruby -e '/regexp/az' -e:1: unknown regexp options - az /regexp/az -e: compile error (SyntaxError) 

Thus, it should probably not be construed as IDENTIFIER token.

Therefore, pm_byte_table has been adapted to accept those invalid regexp option values.
Whether it is a valid regexp option or not is checked by pm_regular_expression_flags_create.
For invalid regexp options, PM_ERR_REGEXP_UNKNOWN_OPTIONS is added to diagnostics.

https://github.com/ruby/prism/commit/d2a6096fcf