# Release notes for version 0.1d, 20230719

```
/******************************************************************************
 * This file is part of The Unicode Tools Of Rexx (TUTOR)                     *
 * See https://rexx.epbcn.com/TUTOR/                                          *
 *     and https://github.com/JosepMariaBlasco/TUTOR                          *
 * Copyright © 2023-2025 Josep Maria Blasco <josep.maria.blasco@epbcn.com>    *
 * License: Apache License 2.0 (https://www.apache.org/licenses/LICENSE-2.0)  *
 ******************************************************************************/
```

Today's release of the Unicode Toys contains a number of substantial enhancements.

* The _Rexx Preprocessor for Unicode_, "rxu". You write a program with the ".rxu" extension; this program is a normal (oo)Rexx program, with some extensions (defined below). Then you enter "rxu programname arguments" on the command line, and the Rexx Preprocessor translates your ".rxu" file into a ".rex" one, and then calls this new ".rex" file with the supplied arguments.
* _The Rexx Tokenizer_. Written in Rexx, it supports Regina, ANSI Rexx, ooRexx, and the corresponding Unicode extensions (defined below). _This is a prototype_.
* New syntactic constructs:

__U-strings__, like "(Woman) (zwj) (Man) (zwj) (Woman) (zwj) (Girl) (Father Christmas)"U. They are __Text__ strings, and are resolved at parse-time (to '👩‍👨‍👩‍👧🎅', in this case).
You can include codepoints using the usual hex notation, or a Unicode name, alias or label between parenthesis. If a codepoint is invalid (i.e., > 10FFF or a surrogate), or if a string between parenthesis does not resolve to a codepoint, a syntax error is raised at parse time.

_Example U-strings:_

```rexx {unicode}
"(Father Christmas)"U = "🎅"      -- An emoji, by name
-- "(Father Christmasx)"U         -- Syntax error at parse time, no codepoint is named "Father Christmasx"
-- "(Father Christmas"U           -- Syntax error at parse time (missing right parenthesis)
"(New line)"U = "0A"X             -- An alias. See http://www.unicode.org/Public/UCD/latest/ucd/NameAliases.txt
"(LF)"U = "0A"X                   -- Another alias
--"<Control-000A>"U = "0A"X       -- 000A has no name, but it has a label
"000A"U = "0A"X                   -- A codepoint
--"110000"U                       -- Syntax error (codepoint > 10FFF)
"DB7F"                            -- Syntax error (a surrogate)
"(LATIN CAPITAL LETTER A)"U = "A" -- The official name for this codepoint
"(Latin capital letter A)"U = "A" -- Case insensitive
"(LATIN-CAPITAL-LETTER-A)"U = "A" -- The standard allows these variations
"(LATINCAPITALLETTERA)"U = "A"    -- And these too
```

__T-strings__, like `"noël👩‍👨‍👩‍👧🎅"T`. They are __Text__ strings, and they are checked at parse time for UTF-8 correctness (i.e., if there is an invalid UTF-8 string, a syntax error is raised).

__R-strings__, like `"noël👩‍👨‍👩‍👧🎅"R`. They are __Runes__ strings, and they are checked at parse time for UTF-8 correctness (i.e., if there is an invalid UTF-8 string, a syntax error is raised).

The RXU preprocessor handles the syntax checking for U-, T- and T-strings, and translates them to "normal" Rexx, assuming the use of Unicode.cls.
For example, `"noël👩‍👨‍👩‍👧🎅"T` is translated to `Text("noël👩‍👨‍👩‍👧🎅")`.

RXU also substitutes several BIF function calls and adds an exclamation mark at the beginning of the function name.
For example, ``l = length(var)`` is translated to ``l = !length(var)``. ``Unicode.cls`` then defines the !-BIFs, which are rerouted to the corresponding BIMs (i.e., to ``Bytes`` [``.String``], ``Runes``,
or ``Text``, as appropiate).

This way you can write ``Length("noël👩‍👨‍👩‍👧🎅")``, and be sure that you will get the correct result (34 for Bytes, 12 for Runes, and 6 for Text), without having to add internal function handlers for Length.

RXU unconditionally adds a last ``::Requires Unicode.cls`` line to the generated ``.rex file``.
The ooRexx processor doesn't complain if there are several of these, and this way we ensure that we have access to the new classes, the !-BIFs and the new BIFs (like ``BYTES()``, ``TEXT()`` or ``RUNES()``).

I would have liked to write some documentation, apart from the one that is included inside the source files, but I have opted to release the code first. That way I will be able to incorporate your comments/suggestions, etc., and I will not have to write the documentation twice.

I am especially interested in your comments about the syntax of the U-Strings, possible extensions, etc.

I am attaching a small sample program and its output below my signature.

Want to give it a try? Just download everything from https://github.com/RexxLA/rexx-repository/tree/master/ARB/standards/work-in-progress/unicode/UnicodeToys (including the subdirectories) and start to experiment.

  Josep Maria

```rexx {unicode}
------------------------- Sample program "sample.rxu" -------------------------
text = "(Woman) (zwj) (Man) (zwj) (Woman) (zwj) (Girl) (Father Christmas)"U

Say "Text is: '"text"'."
Say "It is a" StringType(text) "string."
Say "Its length is" Length(text)"."
Do i = 1 To Length(text)
  Say "  "i":" text[i] "('"c2x(text[i])"'X)"
End
Say "Reversed, it's '"Reverse(text)"'."

Say

Say "Now we will convert the Text string to a Runes string."
text = Runes(text)
Say "Text is: '"text"'."
Say "It is a" StringType(text) "string."
Say "Its length is" Length(text)"."
Do i = 1 To Length(text)
  Say "  "i":" text[i] "('"c2x(text[i])"'X)"
End
Say "Reversed, it's '"Reverse(text)"'."

Say

text = "noël👩‍👨‍👩‍👧🎅"T
Say "'"text"'T is a" StringType(text) "string of length" Length(text)"."

Do i = 8 To 1 By -1
  Say "  Left('"text","i"') = '"Left(text,i)"'"
End

Say

text = "noël👩‍👨‍👩‍👧🎅"R
Say "'"text"'R is a" StringType(text) "string of length" Length(text)"."
Do i = 14 To 1 By -1
  Say "  Left('"text","Right(i,2)"') = '"Left(text,i)"'"
End

Say

text = "noël👩‍👨‍👩‍👧🎅"
Say "'"text"' is a" StringType(text) "string of length" Length(text)"."
---------------------- End of sample program "sample.rxu" ---------------------
```

```
-------------------- Output from the "rxu sample" command ---------------------
Text is: '👩‍👨‍👩‍👧🎅'.
It is a TEXT string.
Its length is 2.
  1: 👩‍👨‍👩‍👧 ('F09F91A9E2808DF09F91A8E2808DF09F91A9E2808DF09F91A7'X)
  2: 🎅 ('F09F8E85'X)
Reversed, it's '🎅👩‍👨‍👩‍👧'.

Now we will convert the Text string to a Runes string.
Text is: '👩‍👨‍👩‍👧🎅'.
It is a RUNES string.
Its length is 8.
  1: 👩 ('F09F91A9'X)
  2: ‍ ('E2808D'X)
  3: 👨 ('F09F91A8'X)
  4: ‍ ('E2808D'X)
  5: 👩 ('F09F91A9'X)
  6: ‍ ('E2808D'X)
  7: 👧 ('F09F91A7'X)
  8: 🎅 ('F09F8E85'X)
Reversed, it's '🎅👧‍👩‍👨‍👩'.

'noël👩‍👨‍👩‍👧🎅'T is a TEXT string of length 6.
  Left('noël👩‍👨‍👩‍👧🎅,8') = 'noël👩‍👨‍👩‍👧🎅  '
  Left('noël👩‍👨‍👩‍👧🎅,7') = 'noël👩‍👨‍👩‍👧🎅 '
  Left('noël👩‍👨‍👩‍👧🎅,6') = 'noël👩‍👨‍👩‍👧🎅'
  Left('noël👩‍👨‍👩‍👧🎅,5') = 'noël👩‍👨‍👩‍👧'
  Left('noël👩‍👨‍👩‍👧🎅,4') = 'noël'
  Left('noël👩‍👨‍👩‍👧🎅,3') = 'noë'
  Left('noël👩‍👨‍👩‍👧🎅,2') = 'no'
  Left('noël👩‍👨‍👩‍👧🎅,1') = 'n'

'noël👩‍👨‍👩‍👧🎅'R is a RUNES string of length 12.
  Left('noël👩‍👨‍👩‍👧🎅,14') = 'noël👩‍👨‍👩‍👧🎅  '
  Left('noël👩‍👨‍👩‍👧🎅,13') = 'noël👩‍👨‍👩‍👧🎅 '
  Left('noël👩‍👨‍👩‍👧🎅,12') = 'noël👩‍👨‍👩‍👧🎅'
  Left('noël👩‍👨‍👩‍👧🎅,11') = 'noël👩‍👨‍👩‍👧'
  Left('noël👩‍👨‍👩‍👧🎅,10') = 'noël👩‍👨‍👩‍'
  Left('noël👩‍👨‍👩‍👧🎅, 9') = 'noël👩‍👨‍👩'
  Left('noël👩‍👨‍👩‍👧🎅, 8') = 'noël👩‍👨‍'
  Left('noël👩‍👨‍👩‍👧🎅, 7') = 'noël👩‍👨'
  Left('noël👩‍👨‍👩‍👧🎅, 6') = 'noël👩‍'
  Left('noël👩‍👨‍👩‍👧🎅, 5') = 'noël👩'
  Left('noël👩‍👨‍👩‍👧🎅, 4') = 'noël'
  Left('noël👩‍👨‍👩‍👧🎅, 3') = 'noë'
  Left('noël👩‍👨‍👩‍👧🎅, 2') = 'no'
  Left('noël👩‍👨‍👩‍👧🎅, 1') = 'n'

'noël👩‍👨‍👩‍👧🎅' is a BYTES string of length 34.
----------------- End of output from the "rxu sample" command -----------------
```
