REX First Public Paper - SHARE 56

 REX -- A Command Programming Language            18 Feb 1981
 =====================================               SHARE 56

      Languages  which  allow you to combine useful sequences
 of commands to create new commands are increasing in  impor-
 tance  in  modern  Operating Systems.   In CMS, for example,
 most commands are issued with the aid of EXECs,  and  editor
 macros increase the power and flexibility of our Editors.
      REX  is  an  experimental  command programming language
 which has the syntax and  structure  of  higher  level  lan-
 guages, yet which maintains the advantages and power of EXEC
 2.    EXECs  are much more readable when written in REX, and
 are more easily maintained.  The powerful expression  evalu-
 ation  and  string parsing facilities can greatly reduce the
 length of EXECs, and the enhanced (and interactive)  tracing
 facilities make debugging easier.

 What is REX?
 REX  is a command programming language.  It can be used as a
 direct replacement for or alternative to the  CMS  EXEC  and
 EXEC  2  languages,  and  as a "Macro processor" for editors

 REX is a language with  useful  control  structures,  string
 parsing,  and  general  expression evaluation.   It also has
 good tracing facilities (including  a  powerful  interactive
 debug  mode),  and  is easy to learn and use.  It seems that
 both "end users" and programmers find REX a simple  and  ef-
 fective language.

 Like  EXEC  2,  REX has several advantages over the original
 CMS EXEC language:  it has considerably  enhanced  function,
 it does not tokenise data, and if the Exec involves loops of
 any kind, then it is significantly faster.

 The language itself is PL/I-like, and essentially system in-
 dependent.    In  its  CMS  implementation it is usually in-
 stalled as an alternative to the standard  processors.    In
 this  mode  one  can  write Execs or Editor Macros in one of
 three languages: EXEC (standard CMS), EXEC 2  (its  replace-
 ment), or REX.

 The  REX  interface  will examine the file and pass it on to
 the appropriate interpreter. (If the file begins with a  REX
 comment  it  will  be interpreted by REX, etc.).  This means
 that REX coexists with both EXEC and EXEC 2  and  users  may
 gradually  convert  to  REX  without having to change any of
 their existing Execs or Macros.   REX can  also  be  invoked
 from a program with the data to be interpreted held in stor-
 age, so avoiding File System overheads.

 Why REX was designed
 The CMS EXEC language (which has since been extended and im-
 proved upon by EXEC 2) is based on the common macro language
 principle  that  variables  and  controls  should be distin-
 guished (by "&") and literals should exist in "plain text".

 When Execs consisted mainly of  strings  of  commands,  with
 very  little  logic in between, this was a fair and sensible
 choice: however a quick scan through the Execs of almost any
 modern user quickly shows that the majority of words in  use
 are  symbolic (that is, they begin with "&").  This observa-
 tion must cast some doubt on the validity of using this syn-

 A further argument is the increasing  use  of  "complicated"
 strings  in  Execs:  for example embedded blanks are heavily
 used in Editor Macros; full  screen  displays;  and  so  on.
 EXEC  2 handles these quite well, but EXEC cannot manipulate
 them at all: the user is reduced to unreadable manipulations
 of the Underscore character or other machinations to achieve
 the desired result.  In REX, blanks can be handled as easily
 as any other character.

 Thirdly,  the  necessity  of  using  upper  case  characters
 throughout the EXEC languages makes them awkward to type and
 difficult to read:  it is clear that programs typed in mixed
 case are, like this paper, easier to follow.

 Finally,  the  underlying syntax of the Exec languages makes
 the efficient interpretation (and, perhaps, compilation)  of
 modern control structures extremely difficult, if not impos-
 sible.    However, such facilities are necessary in order to
 easily enhance and maintain Execs and macros once they  have
 been written.

 Therefore  there is, perhaps, some justification in investi-
 gating an alternative macro command language which uses  the
 "more  conventional"  notation used by the higher level pro-
 gramming languages such as PL/I, Pascal,  ADA,  and  so  on.
 Experience suggests that a language with this type of syntax
 will  be  easier  to learn and use than one which is derived
 from a programmers' "Macro language".  Although REX is espe-
 cially attractive to those who are used to programming, many
 people in IBM who before would not learn a command  or  pro-
 gramming language now use REX.

 The use of the more conventional syntax will naturally cause
 users  to draw comparisons with their usual programming lan-
 guages.  This inevitably will lead them to expect  a  corre-
 sponding  improvement  in  the  facilities  available in the
 command language: this in turn would seem to imply that  the
 interpreter  might be larger and probably slower than either
 EXEC or EXEC 2.  Size (within reason) is not often a problem
 on modern virtual machines,  however  a  severe  performance
 penalty  would  be  unacceptable in most environments.  Some
 effort has therefore been made to ensure good performance.

 During implementation it has been found that REX is of simi-
 lar size to the existing interpreters (currently it is about
 18000 bytes, 10% of which are the error messages).

 The REX interpreter is somewhat slower than EXEC 2 for triv-
 ial operations, but for many tasks it is faster.  It is usu-
 ally very much faster than EXEC.

 What then are the major desirable  features  for  a  command
 macro language?  My choices included:

 1.  Structured  flow  control statements, some equivalent of
     If-then-else,  Do  (Iteration/Until/While)-end,  Select-
     when-end being the most important.

 2.  Free  format,  yet  not requiring a terminator for every

 3.  Literal shorthand: unknown "tokens" assumed  to  be  en-
     closed  in  quotes,  with  a natural concatenation mech-

 4.  Some automatic case translation as required by the host,
     and no requirement that keywords and variable  names  be
     typed in upper case, etc.

 5.  "Complex" expressions (i.e. parentheses, multiple opera-

 6.  Built-in parsing facilities for character strings.

 7.  In line "function" calls to other Execs or Modules.

 8.  No requirement for self-modifying Execs.

 9.  "Peer" communication between Execs and programs.

 These and other features are outlined in the foils that fol-
 low this paper.  <1993: not available now in softcopy--sorry>

   Mike Cowlishaw,
   IBM U.K. Laboratories,
   Hursley Park,
   Winchester,  England.

   Currently on Sabbatical at IBM T.J.Watson Research Center.