Zmodem Protocol Program

Feb 19, 2018 Two sample applications are make using the library: mrz is a ZModem receiver, and msz is a ZModem sender. It derives from the lrzsz project, which is code for executables that send and receive files via XModem, YModem, or ZModem. Specifically, this is Mike Gran's fork of jnavila's fork of Uwe Ohse's fork of Chuck Forsberg's rzsz. ZMODEM software escapes ZDLE, 020, 0220, 021, 0221, 023, and 0223. If preceded by 0100 or 0300 (@), 015 and 0215 are also escaped to protect the Telenet command escape CR-@-CR. The receiver ignores 021, 0221, 023, and 0223 characters in the data stream.

ZMODEM is a file transfer protocol developed by Chuck Forsberg in 1986, in a project funded by Telenet in order to improve file transfers on their X.25 network. In addition to dramatically improved performance compared to older protocols, ZMODEM offered restartable transfers, auto-start by the sender, an expanded 32-bit CRC, and control character quoting supporting 8-bit clean transfers, allowing it to be used on networks that would not pass control characters.

In contrast to most transfer protocols developed for bulletin board systems (BBSs), ZMODEM was not directly based on, nor compatible with, the seminal XMODEM. Many variants of XMODEM had been developed in order to address one or more of its shortcomings, and most remained backward compatible and would successfully complete transfers with 'classic' XMODEM implementations. This list includes Forsberg's own YMODEM.

Zmodem Protocol Specification

ZMODEM eschewed backward compatibility in favor of producing a radically improved protocol. It performed as well or better than any of the high-performance varieties of XMODEM, did so over links that previously didn't work at all, like X.25, or had poor performance, like Telebit modems, and included useful features found in few or no other protocols. ZMODEM became extremely popular on bulletin board systems (BBS) in the early 1990s, becoming a standard as widespread as XMODEM had been before it.

Improvements[edit]

Streaming[edit]

Generally, file transfer protocols break down a file into a series of packets, and then send them one-at-a-time to the receiver. The main portion of the packet, the payload, is a certain number of bytes from the file being sent. After the payload comes a checksum or cyclic redundancy check (CRC) that can be used to determine if the payload was received correctly. If the packet is received correctly, the receiver sends an ACK message and the sender then starts sending the next packet.

The telephone system introduces a small delay known as latency that interferes with this process. Even if the receiver sends the ACK immediately, the delay in the phone lines means there will always be some time before the sender receives it and sends the next packet. As modem speeds increase, this delay represents a larger and larger number of packets that could have been sent during the delay, decreasing the channel efficiency.

XMODEM used 128-bytes payloads with a three-byte header and one-byte checksum for a total of 132 bytes per packet. In the era of 300 bps modems, a packet took about four seconds to send, and typical latencies were on the order of 1⁄10 of a second, so the performance overhead was not significant. As speeds increase the problem becomes more problematic; at 2400 bps a packet takes about 1⁄2 to send, so about 1⁄5 of the available bandwidth is wasted waiting for ACKs. At 9600 bps a packet requires only 0.13 seconds to send, so about 1⁄2 of the bandwidth is wasted.

One solution to this problem is the use of a sliding window. These protocols address latency by allowing the sender to continue sending a number of packets without waiting for an ACK. The number of packets that it allows to continue is the 'window', which was typically between two and sixteen packets in most implementations. A number of new versions of XMODEM with sliding window support appeared in the early 1980s.

Sliding windows are useful for latencies on the order of several packet lengths, which is the case for XMODEM on conventional phone lines. However, it is not enough to address longer latencies found on overseas phone calls or X.25 services such as PC Pursuit, where the latencies are on the order of a second or longer. In other cases, where the reverse channel was much slower than the sending one, as was the case for Telebit or US Robotics modems, even the small number of ACKs might overwhelm the return channel and cause the transfer to pause.

ZMODEM addressed these problems by removing the need for ACKs at all, allowing the sender to send data continually as long as the receiver detected no errors. Only NAKs had to be sent, if and only if there was a problem. Since ZMODEM was often used on links with built-in error correction, like X.25, the receiver would often not send a single message back to the sender. As a result, the system would send the entire file in a continual stream, and ZMODEM referred to itself as a 'streaming protocol'.

ZMODEM's performance was so improved over previous common protocols that it generally replaced even special protocols such as YMODEM-g, which included no error correction at all and instead relied on error-free links maintained by the modems. Although YMODEM-g was faster, the lack of other features such as restartable transfers made it less appealing.

Restart[edit]

XMODEM, and most protocols based on it, managed packet order by prefixing the data with a packet number from 1 to 255. Windowed versions used this packet number to indicate which packets had been received properly, or specify one that had not. Since the packets were 128 bytes long, this meant the maximum amount of data that could be transferred before the packet numbers rolled over was 32 kB.

ZMODEM replaced the packet number with the actual location in the file, indicated by a 32-bit number. This allowed it to send NAK messages that re-wound the transfer to the point of failure, regardless of how long the file might be. This same feature was also used to re-start transfers if they failed or were deliberately interrupted. In this case, the receiver would look to see how much data had been previously received and then send a NAK with that location, automatically triggering the sender to start from that point.

Auto-start[edit]

Auto-starting simplified management by allowing the sending machine to start the transfer. Previously the user had to first request the file from the sender, placing it into a 'waiting' state, then return to their local program and invoke a command to start the transfer. With auto-transfer, they simply requested the file, the sender would then automatically trigger the transfer in the user's program.

Variations[edit]

A number of modified versions of ZMODEM appeared. ZedZap was a variant of ZMODEM with 8 kbyte blocks for better performance on high-speed modems. LeechZmodem was a mischievous ZMODEM variant (among similar XMODEM and YMODEM derivatives) that cheated BBS download quotas. A backwards compatible extension of ZMODEM with 32 kbyte and 64 kbyte block lengths was created by ADONTEC in 2002 and 2007 to increase performance on high-speed error free connections like ISDN or TCP/IP networks.

The most notable ZMODEM implementations were from Chuck Forsberg's Omen Technology, Inc. These included DSZ (DOS Send ZMODEM), GSZ (Graphical Send ZMODEM), and the ubiquitous (l)rzsz for Unix variants.

In more current times, the developers of Synchronet have created a modern X/Y/ZMODEM implementation named SEXYZ, loosely based on the zmtx/zmrx package, which runs natively on Windows and Unix variants, supports long filenames and faster, more reliable data transfers. The ZMODEM implementation from SEXYZ has also been incorporated into the SyncTERM project. Synchronet, SEXYZ, and SyncTERM are all open-source, cross-platform, BBS-centric projects.

Forsberg himself collected a number of improvements into ZMODEM-90. The first of these is MobyTurbo, which removed control quoting to further improve performance, about 15%. Even on networks that 'eat' control characters, ZMODEM-90 can be tailored to quote only those characters the network actually eats, as opposed to every possible one. A similar improvement allows ZMODEM-90 to work on 7-bit networks, whereas earlier protocols (with the notable exception of Kermit) had all demanded 8-bits to one degree or another. Finally, ZMODEM-90 includes a basic run-length encoding compression system to further improve performance on uncompressed files.

Limitations[edit]

• Some of the ZMODEM packets (e.g. ZACK, ZRPOS) embed a byte-offset within the transferred file as a 32-bit unsigned integer. This design limits the feasibility of ZMODEM to only reliably transfer files that are under 4GB in size.
• Even though the protocol could permit it, the reference (l)rzsz implementation cannot encode arbitrary non-control characters (e.g. '~') which are often used by TCP/IP connection programs like telnet and ssh as client-side 'terminal escape' characters. Users must disable the terminal escape feature to achieve reliable transfers over these kinds of links, e.g. ssh -e none user@hostname.

References[edit]

External links[edit]

XMODEM is a simple file transfer protocol developed as a quick hack by Ward Christensen for use in his 1977 MODEM.ASMterminal program. It allowed users to transmit files between their computers when both sides used MODEM. Keith Petersen made a minor update to always turn on 'quiet mode', and called the result XMODEM.^[3]

XMODEM, like most file transfer protocols, breaks up the original data into a series of 'packets' that are sent to the receiver, along with additional information allowing the receiver to determine whether that packet was correctly received. If an error is detected, the receiver requests that the packet be re-sent. A string of bad packets causes the transfer to abort.

XMODEM became extremely popular in the early bulletin board system (BBS) market, largely because it was simple to implement. It was also fairly inefficient, and as modem speeds increased, this problem led to the development of a number of modified versions of XMODEM to improve performance or address other problems with the protocol. Christensen believed his original XMODEM to be 'the single most modified program in computing history'.^[4]

Chuck Forsberg collected a number of common modifications into his YMODEM protocol, but poor implementation led to a further fracturing before they were re-unified by his later ZMODEM protocol. ZMODEM became very popular, but never completely replaced XMODEM in the BBS market.

Packet structure[edit]

The original XMODEM used a 128-byte data packet, the basic block size used on CP/Mfloppy disks. The packet was prefixed by a simple 3-byte header containing a character, a 'block number' from 0-255, and the 'inverse' block number—255 minus the block number. Block numbering starts with 1 for the first block sent, not 0. The header was followed by the 128 bytes of data, and then a single-byte checksum. The complete packet was thus 132 bytes long, containing 128 bytes of payload data, for a total channel efficiency of about 97%.

The checksum was the sum of all bytes in the packet modulo 256. The modulo operation was easily computed by discarding all but the eight least significant bits of the result, or alternatively on an eight-bit machine, ignoring arithmetic overflow which would produce the same effect automatically. In this way, the checksum was restricted to an eight-bit quantity. For example, if this checksum method was used on a tiny data packet containing only two bytes carrying the values 130 and 130, the total of these codes is 260 and the resulting checksum is 4.

The file was marked 'complete' with a character sent after the last block. This character was not in a packet, but sent alone as a single byte. Since the file length was not sent as part of the protocol, the last packet was padded out with a 'known character' that could be dropped. In the original specification, this defaulted to or 26 decimal, which CP/M used as the end-of-file marker inside its own disk format. The standard suggested any character could be used for padding, but there was no way for it to be changed within the protocol itself – if an implementation changed the padding character, only clients using the same implementation would correctly interpret the new padding character.

Transfer details[edit]

Files were transferred one packet at a time. When received, the packet's checksum was calculated by the receiver and compared to the one received from the sender at the end of the packet. If the two matched, the receiver sent an message back to the sender, which then sent the next packet in sequence. If there was a problem with the checksum, the receiver instead sent a . If a was received, the sender would re-send the packet, and continued to try several times, normally ten, before aborting the transfer.

A was also sent if the receiver did not receive a valid packet within ten seconds while still expecting data due to the lack of a character. A seven-second timeout was also used within a packet, guarding against dropped connections in mid-packet.

The block numbers were also examined in a simple way to check for errors. After receiving a packet successfully, the next packet should have a one-higher number. If it instead received the same block number this was not considered serious, it was implied that the had not been received by the sender, which had then re-sent the packet. Any other packet number signalled that packets had been lost.

Transfers were receiver-driven; the transmitter would not send any data until an initial was sent by the receiver. This was a logical outcome of the way the user interacted with the sending machine, which would be remotely located. The user would navigate to the requested file on the sending machine, and then ask that machine to transfer it. Once this command was issued, the user would then execute a command in their local software to start receiving. Since the delay between asking the remote system for the file and issuing a local command to receive was unknown, XMODEM allowed up to 90 seconds for the receiver to begin issuing requests for data packets.

Problems[edit]

Although XMODEM was robust enough for a journalist in 1982 to transmit stories from Pakistan to the United States with an Osborne 1 and acoustic coupler over poor-quality telephone lines,^[5] the protocol had several flaws.

Minor problems[edit]

XMODEM was written for CP/M machines, and bears several marks of that operating system. Notably, files on CP/M were always multiples of 128 bytes, and their end was marked within a block with the character. These characteristics were transplanted directly into XMODEM. However, other operating systems did not feature either of these peculiarities, and the widespread introduction of MS-DOS in the early 1980s led to XMODEM having to be updated to notice either a or as the end-of-file marker.

For some time it was suggested that sending a character instead of an or should be supported in order to easily abort the transfer from the receiving end. Likewise, a received in place of the indicated the sender wished to cancel the transfer. However, this character could be easily 'created' via simple noise-related errors of what was meant to be an or . A double- was proposed to avoid this problem, but it is not clear if this was widely implemented.

Major problems[edit]

XMODEM was designed for simplicity, without much knowledge of other file transfer protocols – which were fairly rare anyway. Due to its simplicity, there were a number of very basic errors that could cause a transfer to fail, or worse, result in an incorrect file which went unnoticed by the protocol. Most of this was due to the use of a simple checksum for error correction, which is susceptible to missing errors in the data if two bits are reversed, which can happen with a suitably short burst of noise. Additionally, similar damage to the header or checksum could lead to a failed transfer in cases where the data itself was undamaged.

Many authors introduced extensions to XMODEM to address these and other problems. Many asked for these extensions to be included as part of a new XMODEM standard. However, Ward Christensen refused to do this, as it was precisely the lack of these features, and the associated coding needed to support them, which led to XMODEM's widespread use. As he explained:

It was a quick hack I threw together, very unplanned (like everything I do), to satisfy a personal need to communicate with some other people. ONLY the fact that it was done in 8/77, and that I put it in the public domain immediately, made it become the standard that it is...

...People who suggest I make SIGNIFICANT changes to the protocol, such as 'full duplex', 'multiple outstanding blocks', 'multiple destinations', etc etc don't understand that the incredible simplicity of the protocol is one of the reasons it survived.

Batch Transfers[edit]

Another problem with XMODEM was that it required the transfer to be user-driven rather than automated. Typically this meant the user would navigate on the sender's system to select the file they wanted, and then use a command to put that system into the 'ready to send' mode. They would then trigger the transfer from their end using a command in their terminal emulator. If the user wanted to transfer another file, they would have to repeat this process again.

For automated transfers between two sites, a number of add-ons to the XMODEM protocol were implemented over time. These generally assumed the sender would continue sending file after file, with the receiver attempting to trigger the next file by sending a NAK as normal at the start of a transfer. When the NAKs timed out, it could be assumed that either there were no more files, or the link was broken anyway.

MODEM7[edit]

MODEM7, also known as MODEM7 batch or Batch XMODEM, was the first known extension of the XMODEM protocol. A normal XMODEM file transfer starts with the receiver sending a single NAK character to the sender, which then starts sending a single SOH to indicate the start of the data, and then packets of data.

MODEM7 changed this behavior only slightly, by sending the filename, in 8.3 filename format, before the . Each character was sent individually and had to be echoed by the receiver as a form of error correction. For a non-aware XMODEM implementation, this data would simply be ignored while it waited for the SOH to arrive, so the characters would not be echoed and the implementation could fall back to conventional XMODEM. With 'aware' software, the file name could be used to save the file locally. Transfers could continue with another , each file is saved under the name being sent to the receiver.

Jerry Pournelle in 1983 described MODEM7 as 'probably the most popular microcomputer communications program in existence'.^[6]

TeLink[edit]

MODEM7 sent the filename as normal text, which meant it could be corrupted by the same problems that XMODEM was attempting to avoid. This led to the introduction of TeLink by Tom Jennings, author of the original FidoNet mailers.

TeLink avoided MODEM7's problems by standardizing a new 'zero packet' containing information about the original file. This included the file's name, size, and timestamp, which were placed in a regular 128 byte XMODEM block. Whereas a normal XMODEM transfer would start with the sender sending 'block 1' with a header, the TeLink header packet was labeled 'block 0' and began with a . The packet contained the file creation date and time, filename up to 16 characters, the file size as a 4-byte value, and the name of the program sending the file.^[7]

A normal XMODEM implementation would simply discard the packet, the assumption being that the packet number had been corrupted. But this led to a potential time delay if the packet were discarded, as the sender could not tell whether the receiver had responded with a because it did not understand the zero packet or because there was a transmission error. As TeLink was normally used only by FidoNet software, which demanded it as part of the FidoNet standards, this did not present a real-world problem as both ends would always support this standard.^[7]

The basic 'block 0' system became a standard in the FidoNet community, and was re-used by a number of future protocols like SEAlink and YMODEM.

XMODEM-CRC[edit]

The checksum used in the original protocol was extremely simple, and errors within the packet could go unnoticed. This led to the introduction of XMODEM-CRC by John Byrns,^[8][9] which used a 16-bit CRC in place of the 8-bit checksum. CRC's encode not only the data in the packet, but its location as well, allowing it to notice the bit-replacement errors that a checksum would miss. Statistically, this made the chance of detecting an error less than 16 bits long 99.9969%, and even higher for longer error bit strings.^[10]

XMODEM-CRC was designed to be backwardly compatible with XMODEM. To do this, the receiver sent a C (capital C) character instead of a to start the transfer. If the sender responded by sending a packet, it was assumed the sender 'knew' XMODEM-CRC, and the receiver continued sending C's. If no packet was forthcoming, the receiver assumed the sender did not know the protocol, and sent an to start a 'traditional' XMODEM transfer.^[10]

Unfortunately, this attempt at backward compatibility had a downside. Since it was possible that the initial C character would be lost or corrupted, it could not be assumed that the receiver did not support XMODEM-CRC if the first attempt to trigger the transfer failed. The receiver thus tried to start the transfer three times with C, waiting three seconds between each attempt. This meant that if the user selected XMODEM-CRC while attempting to talk to any XMODEM, as it was intended, there was a potential 10 second delay before the transfer started.^[10]

To avoid the delay, the sender and receiver would generally list XMODEM-CRC separately from XMODEM, allowing the user to select 'basic' XMODEM if the sender didn't explicitly list it. To the average user, XMODEM-CRC was essentially a 'second protocol', and treated as such. This was not true of FidoNet mailers, however, where CRC was defined as the standard for all TeLink transfers.^[7]

Zmodem Protocol

Higher throughput[edit]

Since the XMODEM protocol required the sender to stop and wait for an or message from the receiver, it tended to be quite slow. In the era of 300 bit/s modems, the entire 132-byte packet required just over 3.5 seconds to send (132 bytes * (8 bits per byte + 1 start bit + 1 stop bit) / 300 bits per second). Assuming it takes 0.2 seconds for the receiver's to make it back to the sender and the next packet to start hitting the receiver (0.1 seconds in both directions), the overall time for one packet would be 3.7 seconds, just over 92% throughput.

As modem speeds increased, the fixed delay needed to send the / grew in proportion to time needed to send the packet. For instance, at 2400 bit/s the packets took only 0.44 seconds to send, so if the / still took 0.2 seconds to make it back (this is latency in the network, not throughput), the throughput has fallen to under 60%. At 9600 bit/s it is under 30% – more time is spent waiting for the reply than is needed to send the packet.

A number of new versions of XMODEM were introduced in order to address these problems. Like earlier extensions, these versions tended to be backward-compatible with the original XMODEM, and like those extensions, this led to a further fracturing of the XMODEM landscape in the user's terminal emulator. In the end, dozens of versions of XMODEM emerged.

WXModem[edit]

WXmodem, short for 'Windowed Xmodem', is a variant of XMODEM developed by Peter Boswell in 1986 for use on high-latency lines, specifically public X.25 systems and PC Pursuit. These have latencies that are far higher than the plain-old telephone service, which leads to very poor efficiency in XMODEM. Additionally, these networks often use control characters for flow control and other tasks, notably XON/XOFF will stop the data flow. Finally, in the case of an error that required a resend, it was sometimes difficult to know whether a SOH was a packet indicator or more noise. WXmodem adapted XMODEM-CRC to address these problems.^[10]

One change was to escape a small set of control characters, DLE, XON, XOFF and SYN. These were escaped by inserting a DLE in front of them, and then modifying the character by XORing it with 64. In theory, this meant the packet might be as long as 264 bytes if it originally consisted entirely of characters that required escaping. These inserted and modified characters are not part of the CRC calculation, they are removed and converted at the receiving end before calculating the CRC.^[10]

Additionally, all packets were prefixed with a SYN character, which meant the packet lead-in was SYNSOH, reducing the chance that a stray SOH would be confused for a packet header in various error cases. An unescaped SYN found in the body of a packet was an error.^[10]

The major change in WXMODEM is the use of a sliding window to improve throughput on high-latency links. To do so, the ACK messages were followed by the packet number they were ACKing or NAKing. The receiver does not have to ACK every packet, it is allowed to ACK any number between one and four packets. An ACK with the fourth packet sequence number is assumed to ACK all four packets. A error causes a NAK to be sent immediately, with all packets from that number and after being re-sent.^[10]

Requiring an ACK every four packets makes the system work like it has a packet size of 512 bytes, but in the case of an error, typically only requires 128 bytes to be re-sent. Moreover, it reduces the amount of data flowing in the reverse direction by four times. This is of little interest in the typical modem's full duplex operation, but is important in half duplex systems like Telebit models which have 19 kB speed in one direction and 75 bits/s in the return channel.

SEAlink[edit]

One of the first third-party mailers for the FidoNet system was SEAdog, written by the same author as the then-popular .arcdata compression format. SEAdog included a wide variety of improvements, including SEAlink, an improved transfer protocol based on the same sliding window concept as WXmodem.^[11] It differed from WXmodem mostly in details.

One difference is that SEAlink supported the 'zero packet' introduced by TeLink, which is needed in order to operate as a drop-in replacement for TeLink in FidoNet systems where the header was expected. ACKs and NAKs were extended to three-byte 'packets', starting with the ACK or NAK, then the packet number, then the complement of the packet number, in the same fashion as the original XMODEM packet header. The window size was normally set to six packets.^[11]

SEAlink was not expected to operate over X.25 or similar links, and thus did not perform escaping. This was also needed so the zero packet would work properly, as this standard used the SYN character that WXmodem had re-purposed.^[11] On top of these changes, it added an 'Overdrive' mode for half duplex links. This suppressed ACKs for packets that were successfully transferred, in effect making the window of infinite size. This mode was indicated by a flag in the zero block.^[11]

SEAlink later added a number of other improvements and was a useful general-purpose protocol. However, it remained rare outside the FidoNet world, and was rarely seen in user-facing software.

XMODEM-1K[edit]

Another way to solve the throughput problem is to increase the packet size. Although the fundamental problem of latency remains, the speed at which it becomes a problem is higher. XMODEM-1K with 1024-byte packets was the most popular such solution. In this case, the throughput at 9600 bit/s is 81%, given the same assumptions as above.

XMODEM-1K was an expanded version of XMODEM-CRC, which indicated the longer block size in the sender by starting a packet with the character instead of . Like other backward-compatible XMODEM extensions, it was intended that a -1K transfer could be started with any implementation of XMODEM on the other end, backing off features as required.

Another problem with XMODEM was that it required the transfer to be user-driven rather than automated. Typically this meant the user would navigate on the sender's system to select the file they wanted, and then use a command to put that system into the 'ready to send' mode. They would then trigger the transfer from their end using a command in their terminal emulator. If the user wanted to transfer another file, they would have to repeat this process again.

For automated transfers between two sites, a number of add-ons to the XMODEM protocol were implemented over time. These generally assumed the sender would continue sending file after file, with the receiver attempting to trigger the next file by sending a NAK as normal at the start of a transfer. When the NAKs timed out, it could be assumed that either there were no more files, or the link was broken anyway.

MODEM7[edit]

MODEM7, also known as MODEM7 batch or Batch XMODEM, was the first known extension of the XMODEM protocol. A normal XMODEM file transfer starts with the receiver sending a single NAK character to the sender, which then starts sending a single SOH to indicate the start of the data, and then packets of data.

MODEM7 changed this behavior only slightly, by sending the filename, in 8.3 filename format, before the . Each character was sent individually and had to be echoed by the receiver as a form of error correction. For a non-aware XMODEM implementation, this data would simply be ignored while it waited for the SOH to arrive, so the characters would not be echoed and the implementation could fall back to conventional XMODEM. With 'aware' software, the file name could be used to save the file locally. Transfers could continue with another , each file is saved under the name being sent to the receiver.

Jerry Pournelle in 1983 described MODEM7 as 'probably the most popular microcomputer communications program in existence'.^[6]

TeLink[edit]

MODEM7 sent the filename as normal text, which meant it could be corrupted by the same problems that XMODEM was attempting to avoid. This led to the introduction of TeLink by Tom Jennings, author of the original FidoNet mailers.

TeLink avoided MODEM7's problems by standardizing a new 'zero packet' containing information about the original file. This included the file's name, size, and timestamp, which were placed in a regular 128 byte XMODEM block. Whereas a normal XMODEM transfer would start with the sender sending 'block 1' with a header, the TeLink header packet was labeled 'block 0' and began with a . The packet contained the file creation date and time, filename up to 16 characters, the file size as a 4-byte value, and the name of the program sending the file.^[7]

A normal XMODEM implementation would simply discard the packet, the assumption being that the packet number had been corrupted. But this led to a potential time delay if the packet were discarded, as the sender could not tell whether the receiver had responded with a because it did not understand the zero packet or because there was a transmission error. As TeLink was normally used only by FidoNet software, which demanded it as part of the FidoNet standards, this did not present a real-world problem as both ends would always support this standard.^[7]

The basic 'block 0' system became a standard in the FidoNet community, and was re-used by a number of future protocols like SEAlink and YMODEM.

XMODEM-CRC[edit]

The checksum used in the original protocol was extremely simple, and errors within the packet could go unnoticed. This led to the introduction of XMODEM-CRC by John Byrns,^[8][9] which used a 16-bit CRC in place of the 8-bit checksum. CRC's encode not only the data in the packet, but its location as well, allowing it to notice the bit-replacement errors that a checksum would miss. Statistically, this made the chance of detecting an error less than 16 bits long 99.9969%, and even higher for longer error bit strings.^[10]

XMODEM-CRC was designed to be backwardly compatible with XMODEM. To do this, the receiver sent a C (capital C) character instead of a to start the transfer. If the sender responded by sending a packet, it was assumed the sender 'knew' XMODEM-CRC, and the receiver continued sending C's. If no packet was forthcoming, the receiver assumed the sender did not know the protocol, and sent an to start a 'traditional' XMODEM transfer.^[10]

Unfortunately, this attempt at backward compatibility had a downside. Since it was possible that the initial C character would be lost or corrupted, it could not be assumed that the receiver did not support XMODEM-CRC if the first attempt to trigger the transfer failed. The receiver thus tried to start the transfer three times with C, waiting three seconds between each attempt. This meant that if the user selected XMODEM-CRC while attempting to talk to any XMODEM, as it was intended, there was a potential 10 second delay before the transfer started.^[10]

To avoid the delay, the sender and receiver would generally list XMODEM-CRC separately from XMODEM, allowing the user to select 'basic' XMODEM if the sender didn't explicitly list it. To the average user, XMODEM-CRC was essentially a 'second protocol', and treated as such. This was not true of FidoNet mailers, however, where CRC was defined as the standard for all TeLink transfers.^[7]

Zmodem Protocol

Higher throughput[edit]

Since the XMODEM protocol required the sender to stop and wait for an or message from the receiver, it tended to be quite slow. In the era of 300 bit/s modems, the entire 132-byte packet required just over 3.5 seconds to send (132 bytes * (8 bits per byte + 1 start bit + 1 stop bit) / 300 bits per second). Assuming it takes 0.2 seconds for the receiver's to make it back to the sender and the next packet to start hitting the receiver (0.1 seconds in both directions), the overall time for one packet would be 3.7 seconds, just over 92% throughput.

As modem speeds increased, the fixed delay needed to send the / grew in proportion to time needed to send the packet. For instance, at 2400 bit/s the packets took only 0.44 seconds to send, so if the / still took 0.2 seconds to make it back (this is latency in the network, not throughput), the throughput has fallen to under 60%. At 9600 bit/s it is under 30% – more time is spent waiting for the reply than is needed to send the packet.

A number of new versions of XMODEM were introduced in order to address these problems. Like earlier extensions, these versions tended to be backward-compatible with the original XMODEM, and like those extensions, this led to a further fracturing of the XMODEM landscape in the user's terminal emulator. In the end, dozens of versions of XMODEM emerged.

WXModem[edit]

WXmodem, short for 'Windowed Xmodem', is a variant of XMODEM developed by Peter Boswell in 1986 for use on high-latency lines, specifically public X.25 systems and PC Pursuit. These have latencies that are far higher than the plain-old telephone service, which leads to very poor efficiency in XMODEM. Additionally, these networks often use control characters for flow control and other tasks, notably XON/XOFF will stop the data flow. Finally, in the case of an error that required a resend, it was sometimes difficult to know whether a SOH was a packet indicator or more noise. WXmodem adapted XMODEM-CRC to address these problems.^[10]

One change was to escape a small set of control characters, DLE, XON, XOFF and SYN. These were escaped by inserting a DLE in front of them, and then modifying the character by XORing it with 64. In theory, this meant the packet might be as long as 264 bytes if it originally consisted entirely of characters that required escaping. These inserted and modified characters are not part of the CRC calculation, they are removed and converted at the receiving end before calculating the CRC.^[10]

Additionally, all packets were prefixed with a SYN character, which meant the packet lead-in was SYNSOH, reducing the chance that a stray SOH would be confused for a packet header in various error cases. An unescaped SYN found in the body of a packet was an error.^[10]

The major change in WXMODEM is the use of a sliding window to improve throughput on high-latency links. To do so, the ACK messages were followed by the packet number they were ACKing or NAKing. The receiver does not have to ACK every packet, it is allowed to ACK any number between one and four packets. An ACK with the fourth packet sequence number is assumed to ACK all four packets. A error causes a NAK to be sent immediately, with all packets from that number and after being re-sent.^[10]

Requiring an ACK every four packets makes the system work like it has a packet size of 512 bytes, but in the case of an error, typically only requires 128 bytes to be re-sent. Moreover, it reduces the amount of data flowing in the reverse direction by four times. This is of little interest in the typical modem's full duplex operation, but is important in half duplex systems like Telebit models which have 19 kB speed in one direction and 75 bits/s in the return channel.

SEAlink[edit]

One of the first third-party mailers for the FidoNet system was SEAdog, written by the same author as the then-popular .arcdata compression format. SEAdog included a wide variety of improvements, including SEAlink, an improved transfer protocol based on the same sliding window concept as WXmodem.^[11] It differed from WXmodem mostly in details.

One difference is that SEAlink supported the 'zero packet' introduced by TeLink, which is needed in order to operate as a drop-in replacement for TeLink in FidoNet systems where the header was expected. ACKs and NAKs were extended to three-byte 'packets', starting with the ACK or NAK, then the packet number, then the complement of the packet number, in the same fashion as the original XMODEM packet header. The window size was normally set to six packets.^[11]

SEAlink was not expected to operate over X.25 or similar links, and thus did not perform escaping. This was also needed so the zero packet would work properly, as this standard used the SYN character that WXmodem had re-purposed.^[11] On top of these changes, it added an 'Overdrive' mode for half duplex links. This suppressed ACKs for packets that were successfully transferred, in effect making the window of infinite size. This mode was indicated by a flag in the zero block.^[11]

SEAlink later added a number of other improvements and was a useful general-purpose protocol. However, it remained rare outside the FidoNet world, and was rarely seen in user-facing software.

XMODEM-1K[edit]

Another way to solve the throughput problem is to increase the packet size. Although the fundamental problem of latency remains, the speed at which it becomes a problem is higher. XMODEM-1K with 1024-byte packets was the most popular such solution. In this case, the throughput at 9600 bit/s is 81%, given the same assumptions as above.

XMODEM-1K was an expanded version of XMODEM-CRC, which indicated the longer block size in the sender by starting a packet with the character instead of . Like other backward-compatible XMODEM extensions, it was intended that a -1K transfer could be started with any implementation of XMODEM on the other end, backing off features as required.

XMODEM-1K was originally one of the many improvements to XMODEM introduced by Chuck Forsberg in his YMODEM protocol. Forsberg suggested that the various improvements were optional, expecting software authors to implement as many of them as possible. Instead, they generally implemented the bare minimum, leading to a profusion of semi-compatible implementations, and eventually, the splitting out of the name 'YMODEM' into 'XMODEM-1K' and a variety of YMODEMs. Thus XMODEM-1K actually post-dates YMODEM, but remained fairly common anyway.

Zmodem Protocol Program Free

NMODEM[edit]

NMODEM is a file transfer protocol developed by L. B. Neal, who released it in 1990. NMODEM is essentially a version of XMODEM-CRC using larger 2048 byte blocks. The block size was chosen to match the common cluster size of the MS-DOSFAT file system on contemporary hard drives, making buffering data for writing simpler.^[12][13]

Protocol spoofing[edit]

Over reliable (error-free) connections, it is possible to eliminate latency by 'pre-acknowledging' the packets, a technique known more generally as 'protocol spoofing'. This is normally accomplished in the link hardware, notably Telebit modems. The modems, when the option was turned on, would notice the XMODEM header and immediately sent an ACK. This would cause the sending XMODEM program to immediately send the next packet, making the transfer continuous, like an infinite-sized window. The modems also suppress the ACK being sent by the XMODEM software at the far end, thereby freeing up the low-speed return channel.

The system can also be implemented in the protocol itself, and variations of XMODEM offered this feature. In these cases, the receiver would send the ACK as soon as the packet started, in the same fashion as the Telebit modems. Since this feature is only an alteration of the receiver-side behavior, it does not require any changes in the protocol on the sender's side. YMODEM formalized this system.

This concept should be contrasted with the one used in SEAlink, which changes the behavior on both sides of the link. In SEAlink, the receiver stops sending the ACK entirely, and the sender changes its behavior to not expect them.

See also[edit]

References[edit]

Citations[edit]

1. ^Telecommunications: XMODEM: A Standard Is Born, By Alfred Glossbrenner, PC Mag, 17 April 1984, Page 451-452, ... but the protocol itself was long ago placed in the public domain by its creator, Chicagoan Ward Christensen. Since its introduction in 1978, XMODEM ...
2. ^In Focus: History lesson: Ward Christensen's free free-exchange software, By Michael Swaine, InfoWorld, 1 Nov 1982, Page 26
3. ^Ward Christensen, 'Memories', 25 November 1992
4. ^'The Virtual Community'.
5. ^Kline, David (July 1982). 'Osborne—Behind Guerrilla Lines'. Microcomputing. pp. 42–50. Retrieved 15 February 2016.
6. ^Pournelle, Jerry (July 1983). 'Interstellar Drives, Osborne Accessories, DEDICATE/32, and Death Valley'. BYTE. p. 334. Retrieved 28 August 2016.
7. ^ ^abcBush 1995, p. G.1.
8. ^Christensen 1982.
9. ^Forsberg 1986.
10. ^ ^abcdefgBoswell 1986.
11. ^ ^abcdSEAlink 1987.
12. ^NMODEM 1.12 program and source code
13. ^NMODEM documentation

Bibliography[edit]

• Bush, Randy (30 September 1995). FidoNet Technical Standard FTS-0001 (Technical report).
• Christensen, Ward (1 January 1982). 'XMODEM Protocol Overview'.
• Forsberg, Chuck (11 September 1986). 'XMODEM/YMODEM PROTOCOL REFERENCE'.
• Boswell, Peter (20 June 1986). 'XMODEM, CRC XMODEM, WXMODEM File Transfer Protocols'. CRC XMODEM.
• SEALINK File Transfer Protocol (Technical report). 24 August 1987.

External links[edit]

• MODEM.ASM, original source code, Ward Christensen, October 10, 1977.
• XMODEM / YMODEM Protocol Reference by Chuck Forsberg, October 10, 1985
• XMODEM / YMODEM Protocol Reference by Chuck Forsberg, June 18, 1988 (document reformatted October 14, 1988) (HTML version with text issues)
• XMODEM / XMODEM-CRC / WXMODEM File Transfer Protocols, synchro.net
• Adontec XMODEM/32k and XMODEM/64k extensions, adontec.com