UMF - The United Message Format Abstract A number of methods and tools are available for defining the format of messages used for signalling protocols. However, many of these methods and tools have been designed for purposes other than message definition, and have been adopted on the basis that they are readily available rather than being ideally suited to the task. This often means that the methods make it difficult to get definitions correct, or result in unnecessary verbosity both in the definition and on the wire. UMF - the United Message Format - has been custom designed for the purpose of message definition. It is thus easy to specify messages in a compact, extensible format that is readily machine manipulated to produce a compact encoding on the wire. 1. Introduction This document defines the UMF message definition language, and the default text encoding method for messages defined in this way. 2. Requirements for Message Definition and Encoding A good message definition method will have the following properties. It is these properties that UMF has been designed to have. Precise Definitions It is important to accurately capture type information in a message definition. Some message definition methods simply capture the name of a parameter without specifying the type of the parameter (e.g. integer, boolean etc). Additionally types like integers need to be constrained to appropriate values. UMF provides this precision of definition. Compact Definitions The message definition should be as compact as possible, but no more compact. While helpful to the inexperienced developer, excessive keywords and other formatting can actually be Copyright Tech-Know-Ware Ltd, 2002. All rights reserved. Page 1 UMF - The United Message Format August 2002 detrimental to the understanding of the experienced developer. UMF adopts a compact C like definition that contains minimal clutter and thus allows the true message structure to be readily seen at a glance. Readily Extensible The message definition and the resultant on the wire encoding need to support extensibility. As part of this, code should be able to pass over parameters that it does not understand without becoming confused. The UMF message definition and encoding allows this. Extensible by Third Parties It often occurs that a protocol is defined by one body and then adopted and modified by another body. In other cases a base protocol may be defined that is then augmented by external profiles. An effective method of allowing a third-party to accurately specify a message definition as deltas to an existing message definition is important in this respect. UMF allows third-parties to specify protocol additions that should not clash with additions made by other third parties. Machine Parsable It is desirable that the message definition be machine readable so that as much of the slog involved in turning a message definition into running code is as automated as possible. This improves time to market and significantly reduces the potential for adding bugs into the code. An UMF definition is in many respects a generalised form of C data structure definition. Therefore it is relatively simple to convert a machine independent UMF definition into a machine dependent C definition and provide all the code to convert from one data representation to another. This process can remove a vast amount of slog. Additionally, the various compilers involved in the process can do a large amount of validating to ensure that the implementation is correct. Simplicity While accurate message definition is important, it is perhaps even more important that the message definition method be intelligible to people that do not have a great deal of time to become gurus in yet another language. Therefore the definition method should be quick and easy to learn. This means that the message definition language must have minimal complexity. As complexity of Copyright Tech-Know-Ware Ltd, 2002. All rights reserved. Page 2 UMF - The United Message Format August 2002 definition and expressiveness are often interrelated, in some cases it is necessary to restrict expressiveness in the interests of simplicity. Additionally, consideration should also be given to the complexity of the required parser, which may favour simplicity of format over absolute message compactness. UMF is based on the 80-20 principle. It is a small language that can accommodate the majority of situations extremely well. There will be times where a UMF representation is sub-optimal in terms of on-the-wire compactness. However, it is felt that on the whole, the gains in simplicity that this enables outweigh these sub-optimalities. Compact On-the-Wire Encoding As a general principle, it is desirable that encoded messages be as compact as possible. This minimises transmission bandwidth, can make processing the messages more efficient, and prevents premature fragmentation of datagrams. Compact messages are also important in the area of mobile devices that have limited memory and possibly transmission bandwidth. This is particularly the case if the information is stored as persistent configuration data rather than being immediately discarded. Also, in many cases, compact messages are easier for developers experienced in the protocol to read than some more verbose types, and it is these developers that should be the primary target for any measure aimed at easing debugging. Given that there are limits to how compactly the actual data in a message can be represented, the compactness of a message is determined largely by the tagging. Existing protocols often use no tagging of data to minimise message size. They also allow for comma separated lists of parameters that have the same meaning rather than requiring each parameter to be separately tagged. Additionally descriptive parameter names are essential to a clear message definition, but tags used in messages are often shorter than is descriptively useful (e.g.

instead of , instead of ). Therefore, it is desirable to be able to define a descriptive name that can be used in code and a tag name that can be used on the wire. UMF accommodates all of these requirements. Flexible Implementation While turnkey solutions are desirable, they are potentially complex to develop, and thus may incur some cost to use, thus making them inaccessible to some. Therefore a range of implementation routes are desirable, from minimal tools / maximum leg work, to maximal tools/minimum leg work. UMF has a number of implementation routes in addition to the compilation route. An UMF definition can be converted into an Copyright Tech-Know-Ware Ltd, 2002. All rights reserved. Page 3 UMF - The United Message Format August 2002 ABNF definition and implemented via that route, or a DOM like tree based parsing method can be used. (Downloadable software for these implementation routes is - or soon will be - available from [1].) Support Easy Application Debugging Ideally the messages on the wire should be in a form that is aid the debugging process. By default UMF uses a text based line format, and is thus readily readable by human developers. Additionally it is also easy to manually generate test messages. With the aid of cb-like tools, it is possible to format messages so that they are more readable than the most compact line representation. Additional tools make it possible to automatically generate test messages and use them as test vectors to test a parser, or validate that manually generated test messages actually conform to the message definition. Nesting of Protocols In some systems messages from one protocol are carried within messages from another protocol (TCP in IP is a simple example, as is HTML in HTTP). The definition and line encoding should allow this. UMF allows this. Flexible On-the-Wire Encoding It is not always possible to anticipate the direction of development so flexibility in the actual wire representation of the messages is desirable. The principal UMF on-the-wire representation in text based. However, an UMF message definition can also be represented using alternate text formats such as XML, and can also be represented in binary. 2.1 That's UMF UMF has been specifically designed to meet all of the above requirements. 3. UMF Messages Definition This section describes how UMF specifies the content of messages. As the syntax is C-like it is felt that many will immediately understand the message definition. For this reason a short example of a message definition is presented before describing the format in detail. The example is also used to give a rough indication of what the formal Copyright Tech-Know-Ware Ltd, 2002. All rights reserved. Page 4 UMF - The United Message Format August 2002 definition describes, and will thus hopefully help with the understanding of the latter. 3.1 Basic Principles of the Message Definition Before presenting an example, and a more formal definition, it may be helpful to describe the basic principles of the message definition format. Following the C language format, the basic format of a parameter definition is: type name Type specifies things like integers, booleans, ASCII strings, Unicode strings and so on. The name is obviously the name of the parameter. Thus a parameter definition might be: int rfc-number ; In addition, a parameter definition can express constraints on the basic type, cardinality (how many instances of the type are valid in a message), and the tag to be used for the value on the wire. For example, an integer may be limited to the values 0 to 255, and an ASCII string may be limited to a maximum size. The fuller format of a parameter will have the form: type name [cardinality] tagging For example: int <1..30000> referenced-rfcs [0..255] as refers ; This defines an integer that can have values between 1 and 30000. The name of the parameter is refereced-rfcs, but is tagged on-the-wire by 'refers'. The parameter can consist of between 0 and 255 instances of the integer in a valid encoding. Two types of compound parameter are also possible, these being 'struct' and 'union'. Having much the same meaning as they have in C, a struct specifies a group of parameters, all of which may be used in a particular instance of the struct. A union similarly specifies a group of parameters, but in this case only one of the parameters can be used in any one instance of the union. An example of a struct is: Copyright Tech-Know-Ware Ltd, 2002. All rights reserved. Page 5 UMF - The United Message Format August 2002 struct rfc-links { int rfc-number; int <1..32000> referenced-rfcs[0..255] as refers ; }; 3.2 An Example Message Definition The following is an example message definition: module com.tech-know-ware.my-example struct my-example { int <0..255> participant-id as ?; Action action as ?; struct my-addition[0..1] as new.tech-know-ware.com plugin { bool tkw-app-capable as ?; }; }; union action { Join join; Message message as msg; void leave; }; struct Join { ascii<0..63> name; }; struct Message { int <0..255> to-delegates[1..127] as to; ascii<0..255> message as msg; [ // Version 2 additions int <0..5> priority; bool acknowledge as ack; ] [ // Version 5 additions ascii<0..16> font-name[0..1] as font; void bold[0..1]; void italic[0..1]; void underlined[0..1] as ul; ] }; The above definition is intended to represent a very crude meeting Copyright Tech-Know-Ware Ltd, 2002. All rights reserved. Page 6 UMF - The United Message Format August 2002 controller. The first construct (my-example) is the root of all messages for the protocol. Each message identifies a participant using an integer in the range 0 to 255, called participant-id. When encoded on the wire, this parameter will be untagged due to the 'as ?' specification. Each message then has an action, which is also untagged. The type of the action parameter is not immediately specified, and instead references the 'Action' definition. The Action definition is a union in which only one of the specified parameters may appear in an instance of the Action construct. This effectively represents a fork in the semantics of any given message. The options within Action can indicate that somebody has joined the meeting, left the meeting, or is sending a message to other delegates. There is no explicit tag for the 'join' and 'leave' options, so these will be tagged on-the-wire by the parameters' names, 'join' and 'leave' respectively. Conversely, an explicit tag for the 'message' parameter is specified, and hence the message option will be tagged by 'msg' on-the-wire. The join parameter also has a referenced definition. Conceptually, when a person joins a meeting, all the other delegates are informed of their name. The name is an ASCII string that has a minimum length of 0 characters and a maximum length of 63 characters. The message option is also a referenced definition. Conceptually, to send a messages, the participant-id is used to identify the sender, and the to-delegates field contains the participant ids of all the people to whom the message is being sent. On-the-wire, the to-delegates parameter will be tagged with 'to'. Between one and 127 instances of the to-delegates parameter may appear in a message. Also, the message itself is included. The message will consist of ASCII characters and can be between 0 and 255 characters long. On-the-wire, the message field will have the tag 'msg'. The priority and acknowledge fields within the message struct have been added in a later version of the protocol. This is indicated by the square brackets in which the parameters are wrapped. Similarly, font-name, and associated parameters have been added in version 5 of the protocol (according to the comment). The reader should already understand enough of the definition language to understand the meaning of these fields. Returning to the 'my-example' root, a third-party has added an extension to the protocol in the form of the 'my-addition' parameter. It is identified as not being part of the base specification by the keyword 'plugin'. On-the-wire, the additional parameter will be identified by the tag 'new.tech-know-ware.com' to differentiate it Copyright Tech-Know-Ware Ltd, 2002. All rights reserved. Page 7 UMF - The United Message Format August 2002 from additions that may be made by other third parties. On-the-wire encoded examples of this message definition are shown in section 4.2. 3.3 Formal Message Definition Syntax There are two types of parameter in UMF, simple types and compound types. The ABNF definition of these is: UMF-parameter = simple-param / compound-param Simple types represent parameters such as integers, booleans etc. The ABNF definition of a simple param is: simple-param = simple-type WS name [ OWS cardinality ] [ WS "as" WS explicit-tag ] [ WS plugin ] ";" where WS represents white space, and OWS represents optional white space. The 'simple-type' represents the type of the parameter. It can have the following forms: simple-type = "void" / "bool" / "ipv4addr" / "ipv6addr" / "date" / "time" / "oid" / integer-type / string-type / bytes-type / embedded-type / const-type / reference where: integer-type = "int" [ OWS "<" range-constraint ">" ] string-type = ( "ascii" / "unquoted-ascii" / "unicode" ) [ OWS "<" length-constraint ">" ] const-type = "const" OWS "<" first-safe-char *( safe-char ) ">" bytes-type = "bytes" [ OWS "<" length-constraint ">" ] embedded-type = "embedded" [ OWS "<" length-constraint ">" ] reference = [ module-name "::" ] name ; Refers to a type defined ; elsewhere range-constraint = constraint length-constraint = constraint Copyright Tech-Know-Ware Ltd, 2002. All rights reserved. Page 8 UMF - The United Message Format August 2002 constraint = [ min-constraint ".." ] max-constraint min-constraint = ["-"] 1*DIGIT max-constraint = ( ["-"] 1*DIGIT / "*" ) In the case of integer-type, the optional constraint specifies the minimum and maximum permissible values that the integer can take. In the case of string-type, the optional constraint specifies the minimum and maximum number of characters that are allowed to appear in a valid encoding. In the case of bytes-type, the optional constraint specifies the minimum and maximum number of bytes that are allowed to appear in a valid encoding. In the constraint syntax, a maximum value '*' means infinite or unbounded. The various types have the following meaning: void A parameter that has no value. This is most useful in unions, and can also be used to represent boolean events wherein the absence of the parameter indicates false, and the presence of the parameter indicates true. It is more useful than you might at first think! bool Can be true or false int An integer value ipv4addr Represents an IPv4 address, but not the port. ipv6addr Represents an IPv6 address, but not the port. date Date according to the Gregorian calendar, with year, month and date. Other calendar types may be constructed from primitive types if required. Copyright Tech-Know-Ware Ltd, 2002. All rights reserved. Page 9 UMF - The United Message Format August 2002 time Represents the time in hours, minutes and seconds. By default the time is adjusted to UTC, unless the time can be guaranteed to have only local significance. oid This is an ASN.1 style Object Identifier. This is primarily included to enable identification of security protocols. ascii A string made up of ASCII characters, limited at most to values 0 to 127. unquoted-ascii An ascii string usually has quote marks around it. This type does not have quotes around it. Consequently it can not have any white space, or include any special characters (such as "=", "{", and "}") that would confuse the parser. unicode A string made up of Unicode characters. const This type allows a constant value to be inserted into the encoded message. It will typically be untagged. One thing it might be used for is identifying the protocol of the message definition. For example: const protocol as ?; bytes An array of bytes. Also useful for carriage of opaque data. embedded The value is an embedded UMF message. This allows layering of message definitions. The name is the name of the parameter. If there is no explicitly defined tag, then this is also used as the parameter's tag on-the-wire. It has the format: name = ALPHA *( ALPHA / DIGIT / "-" / "_" ) The cardinality of a parameter specifies how many times a particular Copyright Tech-Know-Ware Ltd, 2002. All rights reserved. Page 10 UMF - The United Message Format August 2002 parameter can appear in a message. The format mirrors a C-like array specification, but uses UML style ranges rather than singular values as are required in C. If the cardinality field is absent, then one and only one instance of the parameter must occur in a valid message. The format of the cardinality specification is: cardinality = "[" [ min-occurrences ".." ] max-occurrences "]" min-occurrences = ["-"] 1*DIGIT max-occurrences = ( ["-"] 1*DIGIT / "*" ) Once again, the '*' in max-occurrences represents infinite or unbound. Example cardinalities are as follows: [0..1] ; Zero or one time [0..*] ; Zero or more times [*] ; Same as above, zero or more times [1..*] ; One or more times [5] ; Exactly five times An explicit tag can be any sequence of characters that do not have special significance to the parser. If the tag definition begins with a "?", the "?" is discarded. Thus to specify that ? be used as the tag on-the-wire, specify explicit-tag to be ??. explicit-tag = tag ; tag defined in common definitions Marking an item as plugin indicates to the developer and the tools that this parameter is (probably) not part of the original message definition. For example, it might be a proprietary extension. It also indicates that the parameter may not be present in all received messages, and impacts on the way the binary encoding operates. The compound types are struct and union. For a struct, subject to the various parameters cardinality specifications, any all or none of the parameters that a struct groups together may appear in a valid encoding of the construct. In the case of a union, only one of the parameters may be encoded in a valid instance of the construct. The format of the compound types is similar to the simple types. They have the form: Copyright Tech-Know-Ware Ltd, 2002. All rights reserved. Page 11 UMF - The United Message Format August 2002 compound-param = struct-param / union-param struct-param = "struct" WS name [ OWS cardinality ] [ WS "as" WS explicit-tag ] [ WS pluggable ] [ WS plugin ] OWS "{" struct-body "}" OWS ";" union-param = "union" name [ OWS cardinality ] [ WS "as" WS explicit-tag ] [ WS pluggable ] [ WS plugin ] OWS "{" union-body "}" OWS ";" In a struct and union the pluggable keyword indicates that the construct is a location that the message designers have formally declared as extendible using the 'plug' mechanism that is described further below. UMF compilers are encouraged to emit warnings when extra material has is plugged into locations that are not marked as pluggable, but should not consider it an error. The format of the struct body is: struct-body = *( untagged-UMF-parameter ) *( UMF-parameter ) *( struct-extension ) The struct body starts with all the untagged parameters. Untagged parameters may have a cardinality other than one. Note that, if the cardinality of an untagged parameter allows it to be absent, then when encoded on the wire, all parameters, including tagged parameters must also be absent. Thus great care recommended when defining a message syntax that allows for an untagged parameter to be absent. Following the untagged parameters, the tagged parameters are included. When the message definition is subsequently extended, another instance of the extension parameters construct is added for each version in which the construct is extended. (Note that all new parameters must always be added onto the end of an existing construct, and the order of parameters must never be rearranged from one version to the next.) All of these have a similar format to the types already defined, except that in some cases they may be untagged, or only allow a unary cardinality. To make the ABNF definition accurate it is therefore necessary to repeat the above basic definitions with the appropriate tagging and cardinality specifications. As mentioned, the struct body may start with untagged-UMF-parameters. These are untagged, and must have a cardinality of 1. There definition is: Copyright Tech-Know-Ware Ltd, 2002. All rights reserved. Page 12 UMF - The United Message Format August 2002 untagged-UMF-parameter = untagged-simple-param / untagged-compound-param untagged-simple-type = simple-type WS name [ OWS cardinality ] WS "as" WS "?" OWS ";" untagged-compound-param = untagged-struct-param / untagged-union-param untagged-struct-param = "struct" WS name [ OWS cardinality ] WS "as" WS "?" [ WS pluggable ] OWS "{" struct-body "}" OWS ";" untagged-union-param = "union" WS name [ OWS cardinality ] WS "as" WS "?" [ WS pluggable ] OWS "{" union-body "}" OWS ";" Note that the plugin keyword is not applicable to untagged items. The second part of a struct definition are the items that are tagged. These can have any desired cardinality. These have the basic parameter definition that was initially presented, i.e. UMF-parameter. The third and final part of a struct body is the extension fields. These are parameters that are added in subsequent versions of the protocol specification. They are marked out separately because a parser must always consider absence of these parameters to be a valid encoding so that it can receive messages from entities that are working with an earlier version of the protocol. To do this would dictate that all extension parameters would have to have a cardinality specification that included zero. This is tedious, potentially error prone, and loses some expressiveness. Instead, extension parameters are wrapped inside square brackets to indicate that they are extensions. It is then clear to any tools and developers that these parameters may be absent if a message is received from a host running an earlier version of the message definition. The format of the struct extension is: struct-extension = "[" 1*( UMF-parameter ) "]" The definition of a union-body is as follows: union-body = [ integer-type WS name WS "as" WS "?" OWS ";" ] *( singular-UMF-parameter ) *( union-extension ) A union-body may have a single untagged integer parameter. All other Copyright Tech-Know-Ware Ltd, 2002. All rights reserved. Page 13 UMF - The United Message Format August 2002 parameters must be tagged and have a cardinality of one and only one. A union is extended in much the same way as a struct. The untagged integer parameter allows integers to be defined that have wild-carding options. For example, a union might be defined as: union select { int<0..65535> numbered as ?; void any as *; }; Examples of the encoded form might be: select = 12 select = * The parameters within a union are only allowed unary cardinality to avoid ambiguity in the line encoding. If multiple instances of a parameter must be included as an option in a union, it is necessary to wrap the parameters within a struct, using something similar to: struct X { X x[1..*] as ?; }; As mentioned, most of the parameters within a union are tagged and have a cardinality of one. There defininition is: singular-UMF-parameter = singular-simple-param / singular-compound-param singular-simple-param = simple-type WS name [ WS "as" WS explicit-tag ] [ WS plugin ] OWS ";" singular-compound-param = singular-struct-param / singular-union-param singular-struct-param = "struct" WS name [ WS "as" WS explicit-tag ] [ WS pluggable ] [ WS plugin ] OWS "{" struct-body "}" OWS ";" singular-union-param = "union" WS name [ WS "as" WS explicit-tag ] [ WS pluggable ] [ WS plugin ] OWS "{" union-body "}" OWS ";" The union extension operates in a similar fashion to that of the struct, but references singular-UMF-parameters. Its definition is: Copyright Tech-Know-Ware Ltd, 2002. All rights reserved. Page 14 UMF - The United Message Format August 2002 union-extension = "[" 1*( singular-UMF-parameter ) "]" It was mentioned previously that unions and structs could reference types that are defined elsewhere. The format of a referenced type can now be defined. Referenced types have a cardinality of one, and are untagged. This is because the cardinality and tagging of the type are defined in the item that does the referencing, rather than where the referenced type is defined. (If a referenced type needs a cardinality other than one, it is recommended that the trick for giving a parameter within a union a non-unary cardinality be used.) The definition of the referenced types are: referenced-UMF-parameter = referenced-simple-param / referenced-compound-param referenced-simple-param = simple-type WS name ";" referenced-compound-param = referenced-struct-param / referenced-union-param referenced-struct-param = "struct" WS name [ WS pluggable ] OWS "{" struct-body "}" OWS ";" referenced-union-param = "union" WS name [ WS pluggable ] OWS "{" union-body "}" OWS ";" A protocol may be extended by a third party without modifying the original definition. This may be due to a proprietary extension, or an externally defined profile of the base protocol. The specification for this type of extension is: third-party-extension = "plug" WS tp-struct-extension / tp-union-extension "into" WS name *( "::" name ) *( COMMA name *( "::" name ) ) OWS ";" tp-struct-extension = UMF-parameter tp-union-extension = singular-UMF-parameter This specifies a parameter that is to be plugged into an existing construct. For example, if the following were defined: plug ascii cookie as cookie.tkwumf.com into my-example::my-addition; Copyright Tech-Know-Ware Ltd, 2002. All rights reserved. Page 15 UMF - The United Message Format August 2002 The resulant definition would be treated as if it were: struct my-example { int <0..255> participant-id as ?; Action action as ?; struct my-addition[0..1] as tech-know-ware.com plugin; { bool tkw-app-capable as ?; ascii cookie as cookie.tkwumf.com plugin; }; }; The name field indicates that name of the construct that the item is to be plugged into. A single protocol may be defined in number of message definition file. This might be for the purpose of accessing predefined libraries, or specifying the definition that the current definition extends. A message definition therefore begins with a set of optional directives expressing this information. They have the form: UMF-directive = OWS [ "module" WS module-name WS ] [ "extends" WS module-name OWS ";" OWS ] *( "imports" WS module-name OWS ";" OWS ) module-name = name *( "." name ) Module specifies the name of the module. Extends is used for a definition that contains a third party extension. The module-name in the extends specification indicates the message definition that is being extended. The imports statement indicates a library message definition that contains referenced types that are referenced within the message definition. The module-name follows the hierarchical format used in Java. It is based on a domain name that is created from the name of the protocol, combined with the domain name of the entity that defined it. For example, if a protocol called the Simple Conference Protocol (SCP) were defined by Tech-Know-Ware Ltd with a domain name of tech-know-ware.com, the module name might be: com.tech-know-ware.scp UMF defines a number of pseudo top level domains for its own purposes. These are currently as follows: Copyright Tech-Know-Ware Ltd, 2002. All rights reserved. Page 16 UMF - The United Message Format August 2002 +ietf A pseudo top level domain for the Internet Engineering Task Force. +iso A pseudo top level domain for the International Standards Organisation. The sub-domains of this domain follow the structure of ISO defined Object Identifiers. (All spaces must be removed and numbers in brackets should be ignored when parsing this domain. E.g. iso(1) member-body(2) us(840) rsadsi(113549) digestAlgorithm(2) 5 shall be represented as +iso(1).member-body(2).us(840).rsadsi(113549).digestAlgorithm(2).5 and looked up as +iso.member-body.us.rsadsi.digestAlgorithm.5) +itu A pseudo top level domain for the International Telecommunications Union. The sub-domains of this domain follow the structure of ITU defined Object Identifiers. Processing of such identifiers follows that defined for processing ISO Object Identifiers. +umf A pseudo top level domain for defining UMF extensions and libraries. +uuid A pseudo top level domain that uses Universally Unique Identifiers for identification. An example is: +uuid.4d36e96c-e325-11ce-bfc1-08002be10318 National standards bodies such as ANSI and BSI are defined under their national top-level domain. Finally, we are in a position to describe a complete UMF message definition. This is: UMF-definition = UMF-directives 1* ( referenced-UMF-parameter / third-party-extension ) The first parameter defined within the message definition is the root of the message definition tree, and is thus the outer-most construct of an encoded message. 3.4 Complete ABNF This section presents the complete ABNF of a message definition without narrative. Some definitions are common with the on-the-wire ABNF and a presented in a separate section. UMF-definition = UMF-directives 1* ( referenced-UMF-parameter / third-party-extension ) UMF-directive = OWS Copyright Tech-Know-Ware Ltd, 2002. All rights reserved. Page 17 UMF - The United Message Format August 2002 [ "module" WS module-name WS ] [ "extends" WS module-name OWS ";" OWS ] *( "imports" WS module-name OWS ";" OWS ) module-name = name *( "." name ) referenced-UMF-parameter = referenced-simple-param / referenced-compound-param referenced-simple-param = simple-type WS name ";" simple-type = "void" / "bool" / "ipv4addr" / "ipv6addr" / "date" / "time" / "oid" / integer-type / string-type / bytes-type / embedded-type / const-type / reference integer-type = "int" [ OWS "<" range-constraint ">" ] string-type = ( "ascii" / "unquoted-ascii" / "unicode" ) [ OWS "<" length-constraint ">" ] bytes-type = "bytes" [ OWS "<" length-constraint ">" ] const-type = "const" OWS "<" first-safe-char *( safe-char ) ">" embedded-type = "embedded" [ OWS "<" length-constraint ">" ] reference = [ module-name "::" ] name ; Refers to a type ; defined elsewhere range-constraint = constraint length-constraint = constraint constraint = [ min-constraint ".." ] max-constraint min-constraint = ["-"] 1*DIGIT max-constraint = ( ["-"] 1*DIGIT / "*" ) name = ALPHA *( ALPHANUM / "-" / "_" ) referenced-compound-param = referenced-struct-param / referenced-union-param referenced-struct-param = "struct" WS name [ WS pluggable ] OWS "{" struct-body "}" OWS ";" struct-body = *( untagged-UMF-parameter ) *( UMF-parameter ) *( struct-extension ) referenced-union-param = "union" WS name [ WS pluggable ] OWS "{" union-body "}" OWS ";" union-body = [ integer-type WS name WS "as" WS "?" OWS ";" ] *( singular-UMF-parameter ) Copyright Tech-Know-Ware Ltd, 2002. All rights reserved. Page 18 UMF - The United Message Format August 2002 *( union-extension ) untagged-UMF-parameter = untagged-simple-param / untagged-compound-param untagged-simple-type = simple-type WS name [ OWS cardinality ] WS "as" WS "?" ";" untagged-compound-param = untagged-struct-param / untagged-union-param untagged-struct-param = "struct" WS name [ OWS cardinality ] WS "as" WS "?" [ WS pluggable ] OWS "{" struct-body "}" OWS ";" untagged-union-param = "union" WS name [ OWS cardinality ] WS "as" WS "?" [ WS pluggable ] OWS "{" union-body "}" OWS ";" UMF-parameter = simple-param / compound-param simple-param = simple-type WS name [ OWS cardinality ] [ WS "as" WS explicit-tag ] [ WS plugin ] ";" cardinality = "[" [ min-occurrences ".." ] max-occurrences "]" min-occurrences = ["-"] 1*DIGIT max-occurrences = ( ["-"] 1*DIGIT / "*" ) explicit-tag = tag ; tag defined in common definitions compound-param = struct-param / union-param struct-param = "struct" WS name [ OWS cardinality ] [ WS "as" WS explicit-tag ] [ WS pluggable ] [ WS plugin ] OWS "{" struct-body "}" OWS ";" union-param = "union" WS name [ OWS cardinality ] [ WS "as" WS explicit-tag ] [ WS pluggable ] [ WS plugin ] OWS "{" union-body "}" OWS ";" struct-extension = "[" 1*( UMF-parameter ) "]" singular-UMF-parameter = singular-simple-param / singular-compound-param Copyright Tech-Know-Ware Ltd, 2002. All rights reserved. Page 19 UMF - The United Message Format August 2002 singular-simple-param = type WS name [ WS "as" WS explicit-tag ] [ WS plugin ] ";" singular-compound-param = singular-struct-param / singular-union-param singular-struct-param = "struct" WS name [ WS "as" WS explicit-tag ] [ WS pluggable ] [ WS plugin ] "{" struct-body "}" ";" singular-union-param = "union" WS name [ WS "as" explicit-tag ] [ WS pluggable ] [ WS plugin ] "{" union-body "}" ";" third-party-extension = "plug" WS tp-struct-extension / tp-union-extension "into" WS name *( "::" name ) *( "," name *( "::" name ) ) ";" tp-struct-extension = UMF-parameter tp-union-extension = singular-UMF-parameter 4. On-the-Wire Representation 4.1 Principles of On-the-Wire Encoding The basic format of the text based on-the-wire encoding is to use the format: tag = value If there are multiple instances of a parameter, then they may either be conveyed as multiple instances of the above construct, and as a comma separated list, as in: tag = value, value, value If a tag is explicitly specified in the message definition, then this is used on the wire. If no tag is explicitly specified, then the name of the parameter is used as the tag. It is also possible to explicitly specify that no tag should be used on the wire by setting the explicit tag field to '?'. All untagged items must appear in a struct in the same order that they are defined in the message definition, and must appear before any tagged items within a struct definition. Untagged parameters that have greater Copyright Tech-Know-Ware Ltd, 2002. All rights reserved. Page 20 UMF - The United Message Format August 2002 than one instance must be constructed as a comma separated list. In these cases, the format on the wire becomes: value or: value, value, value If an untagged parameter has a cardinality that allows it to be absent from an encoded message, then all subsequent parameters in the enclosing struct, including tagged parameters, must also be absent. Consequently, great care should be taken when defining a message definition that allows untagged parameters to be absent. Thus, for the examples quoted earlier, that is: int rfc-number ; int <1..30000> referenced-rfcs [0..255] as refers; The format on the wire would be something like (depending on the actual values in question): rfc-number = 3024 refers = 822, 791, 2543 4.2 Example On-the-Wire Representation The following are example on-the-wire representations of the example message. 1 join = { 'Alice' } tech-know-ware.com = { True } 1 msg = { to = 2, 5, 8, 58 msg = 'Where are we going for dinner' } 1 leave 4.3 Formal On-the-Wire Representation The principle representation of an UMF defined message on the wire is text based. Parameters may be untagged as long as they appear before any other tagged parameters. Untagged parameters that have non-singular cardinality must be comma separated. Copyright Tech-Know-Ware Ltd, 2002. All rights reserved. Page 21 UMF - The United Message Format August 2002 The top-level construct of an UMF definition is a referenced type, which essentially has no tag associated with it. (Indeed, the presence of such a tag would not convey any information.) The top-level construct is therefore either a struct body, a union body, or a simple value, as in: UMF-text-message = ( struct-body / union-body ) A struct body can contain untagged and tagged parameters. All untagged parameters must appear before any tagged parameters. The definition of a struct-body is therefore: struct-body = OWS *( value *( COMMA value ) WS ) *( ( tag WS ) / ; For a void parameter ( tag EQUAL value *( COMMA value ) WS ) ) ; WS not required if it's the last item All items of a union body must be tagged, except for a single integer parameter that may be untagged. Also, parameters must only have a cardinality of one in the encoding to avoid ambiguities in the encoded message. Therefore a union body has the form: union-body = OWS (integer-value WS / tag WS / ; For a void parameter ( tag EQUAL value WS ) ) where: value = simple-value / compound-value simple-value = bool-value / integer-value / oid-value / ipv4addr-value / ipv6addr-value / ascii-value / unquoted-ascii-value / unicode-value / const-value / embedded-value / bytes-value / date-value / time-value bool-value = "True" / "False" / "T" / "F" int-value = [ "-" ] 1*DIGIT oid-value = 1*DIGIT *( "~" 1*DIGIT ) ; Only the oid's numerical parts are represented ipv4addr-value = 1*3DIGIT "." 1*3DIGIT "." 1*3DIGIT "." 1*3DIGIT ipv6addr-value = ( 1*4HEX *( ":" 1*4HEX ) [ ":" *( ":" 1*4HEX ] ) Date and time parameters have fixed width to aid parsing. As such Copyright Tech-Know-Ware Ltd, 2002. All rights reserved. Page 22 UMF - The United Message Format August 2002 the various fields have leading zeros if required. Dates are according to the Gregorian calendar. Other calendar types may be constructed from primitive types if required. Typically the time should be converted to UTC prior to including in a message, unless the time can be guaranteed to have only local significance. date-value = date-year "-" date-month "-" date-day date-year = 4DIGIT ; e.g. 2002 date-month = 2DIGIT ; With leading zeros, e.g. 02 date-day = 2DIGIT ; With leading zeros, e.g. 02 time-value = time-hours ":" time-minutes ":" time-seconds time-hours = 2DIGIT ; With leading zeros, e.g. 02 time-minutes = 2DIGIT ; With leading zeros, e.g. 02 time-seconds = 2DIGIT ; With leading zeros, e.g. 02 ; Uses 24 hour clock notation ; All times presented in UTC ascii-value = "'" *( %x00-26 / %x28-5B / %x2D-x7F / "\\" / "\'" ) "'" unquoted-ascii-value = first-safe-char *( safe-char ) unicode-value = DQUOTE *( %x00-21 / %x23-5B / %x5D-xFF / "\\" / "\" DQUOTE ) DQUOTE ; DQUOTE defined in RFC 2234 bytes-value = "^" BASE64 BASE64 = *( 4BASE64-CHAR ) ( ( 4BASE64-CHAR ) / ( 3BASE64-CHAR "=" ) / ( 2BASE64-CHAR "=" "=" ) ) BASE64-CHAR = ALPHA / DIGIT / "+" / "/" const-value = first-safe-char *( safe-char ) embedded-value = "(" *(%x00-28 / %x2A-5B / %x5D-FF / "\)" / "\\" ) ")" ; "\" & ")" are escaped Illustrating the recursiveness of the message format, we have: compound-value = struct-value / union-value Copyright Tech-Know-Ware Ltd, 2002. All rights reserved. Page 23 UMF - The United Message Format August 2002 struct-value = "{" struct-body "}" union-value = union-body EQUAL = OWS "=" OWS COMMA = OWS "," OWS 4.4 Marking Message Boundaries Before a message is parsed it is necessary to know the boundaries of the message. There are many ways in which this can be done, and the method adopted should be specified in the protocol specification. However, in the absence of any other way, UMF parsers should take the presence of an unmatched closing brace to be the end of message marker. Hence, the definition of a message delimited in this way becomes: delimited-UMF-text-message = UMF-text-message "}" 4.5 Illustration of Encoded Types This section illustrates how the types look once they have been encoded according to the syntax above. The tag of each item has the format 'my-XXXX'. Except in the case of the 'void' example, the XXXX part indicates the type that is encoded to the right of the equals sign. my-void // Tag only for a void parameter my-bool = True my-int = 5643 my-ipv4addr = 10.0.0.1 my-ipv6addr = 201:123::0 my-date = 2002-02-28 my-time = 12:00:00 my-oid = 1~2~840~113549~2~5 my-ascii = 'UMF' my-unquoted-ascii = UMF my-unicode = "UMF" my-const = UMF Copyright Tech-Know-Ware Ltd, 2002. All rights reserved. Page 24 UMF - The United Message Format August 2002 my-bytes = ^01AF3C== my-embedded = ( my-other-int=5 single-closing-bracket-text='\)' ) my-struct = { 5434 All time=98787654654 } my-union = 5434 my-union1 = Switch my-union2 = Volume = 11 5. Common ABNF Definitions The following definitions are common to both the definition syntax and the on the wire representation. tag = [ "?" ] first-tag-safe-char *( safe-char ) first-tag-safe-char = %x21 / ; Not " %x23-26 / ; Not ' ( ) %28-2B ; Not , - %x2E-2F / ; Not 0 1 2 3 4 5 6 7 8 9 %x3A-3C / ; Not = %x3E-5D ; Not ^ %x5F-7A / ; Not { %7C / ; Not } %7E-7F ; Visible characters except = , " ' { } ( ) ^ - ; and digits first-safe-char = first-tag-safe-char / DIGIT safe-char = first-safe-char / DQUOTE / "'" / "{" / "(" / "-" / "^" ; Not = } ) , OWS = [ WS ] ; Optional white space WS = comment / " " / HTAB / CR / LF ; HTAB, CR, LF defined in RFC-2234 ; White space may appear between any ; token and is not limited to where ; it is explicitly specified Copyright Tech-Know-Ware Ltd, 2002. All rights reserved. Page 25 UMF - The United Message Format August 2002 comment = c-comment / cpp-comment c-comment = "/*" "*/" cpp-comment = "//" *( HTAB / %x20-%7f ) ( CR / LF ) ; A comment is treated as a single space for the ; purposes of parsing 6. Why UMF The name UMF is pronounced in the same way as 'oomph'. The Collins Paperback English Dictionary (1986) defines oomph as: oomph - (umf) n. Inf. 1. enthusiasm, vigour, or energy. 2. sex appeal. So who wants their code to have UMF? 7. References [UMFHOME]http://www.tech-know-ware.com/umf [ABNF]D. Crocker, & P. Overell, "Augmented BNF for Syntax Specifications: ABNF, " Internet Engineering Task Force, RFC 2234, November 1997. [XML] "Extensible Markup Language (XML) 1.0 (Second Edition)", W3C REC-xml, October 2000. 8. Author's Address Pete Cordell Tech-Know-Ware Ltd P.O. Box 30 Ipswich, IP5 2WY UK pete@tech-know-ware.com 9. ToDo: 10. Changes: ­ Byte array starts with ^. May end with = ­ Struct may have many untagged parameters at the start that have higher than unary cardinality. ­ Added the pluggable keyword to formally mark locations that are Copyright Tech-Know-Ware Ltd, 2002. All rights reserved. Page 26 UMF - The United Message Format August 2002 intended to be externally extendible. For Version 3 ­ Changed byte-array to bytearray For Version 4 ­ Changed bytearray to bytes ­ Added support for OIDs Copyright Tech-Know-Ware Ltd, 2002. All rights reserved. Page 27