/* VTAPE.TXT - KLH10 Virtual Tape Format */ /* $Id: vtape.txt,v 2.3 2001/11/10 21:24:21 klh Exp $ */ /* Copyright © 1994-1999, 2001 Kenneth L. Harrenstien ** All Rights Reserved ** ** This file is part of the KLH10 Distribution. Use, modification, and ** re-distribution is permitted subject to the terms in the file ** named "LICENSE", which contains the full text of the legal notices ** and should always accompany this Distribution. */ One of the methods used by the KLH10 to emulate PDP-10 magnetic tape devices is to provide support for "virtual tapes". These virtual tapes are in reality disk files on the native host system which have a specific format. Because they are native disk files, it is far easier and faster to manipulate these virtual tape images (for reading, writing, copying, archival, and transport over networks) than would be the case for real physical tapes. The purpose of this description is to allow users to generate or maintain virtual tapes using their own tools in addition to the ones provided in the KLH10 distribution. In most cases these tools can be nothing more than a text editor and a simple copy program such as Unix "dd". VIRTUAL TAPE FORMATS ==================== Several virtual tape formats exist, developed more or less independently. The KLH10 software can handle most of them and others could be added if necessary. These formats are: "RAW" - original KLH10 format "TPS" - Wilson & Supnik format "TPC" - DECUS/Shoppa format "ITSDUMP" - special KLH10 format for ITS "RAW" VIRTUAL TAPE: Directory and Data files ================== Each RAW-format virtual tape consists of two native files, one for the actual tape data and the other to describe that data. For example, a virtual tape with the name "kosh" would consist of these two files: Virtual tape "kosh": kosh - Tape directory (text; defines structure of data) kosh.tap - Tape data (binary; one byte per tape frame) "RAW" DATA FILE FORMAT (.TAP) The tape data file is simply a continuous sequence of 8-bit bytes corresponding to the logical stream of 8-bit tape data frames. There is nothing in this file except tape data; there are no record boundaries or tapemarks. Because this is a binary data file, all operations on this file must preserve the data exactly. "RAW" DIRECTORY FILE FORMAT (.TDR) The tape directory is a text file that describes the contents of its associated tape data file; it defines how the tape is structured in terms of record boundaries, tapemarks, length, and internal format. It can be used to describe anything about the tape in an easily readable and editable form. The directory is formatted as a set of logical text lines. Each line begins with a keyword, optionally followed by data. Comments are introduced by a semi-colon character (';') and continue to the end of that physical line. Blank lines are ignored. Whitespace is also ignored except at the start of a line, where indentation may be used only when continuing a logical line over several physical lines. Keywords: TF-Format: [] ; Tape datafile format and optional filename <#>: ; File/Record descriptors ; optional continuation of above EOT: ; Physical End of Tape, remaining text ignored. - REQUIRED. Describes how the tape file data is stored on disk. Currently only one keyword is fully supported: "raw" - 8-bit bytes, one per frame. - DEPRECATED. Optional name of tape data file, a feature no longer used (ie ignored on input, not generated on output). The tape data filename is either specified at mount time or dynamically generated using ".tap" as an extension to the name of the tape directory file. This extension replaces ".tdr" if it exists, otherwise ".tap" is simply appended. <#> - location in data file of 1st record beginning next file. For "raw" format this corresponds to the logical frame # on tape as well, but for other formats this may serve a realignment purpose. The string "BOT" is considered equivalent to "0". - a list of record descriptors composing the file, separated by whitespace. The list may be empty, as is normally the case for two consecutive tapemarks that signal a logical EOT. - A record descriptor, with the format [*][E[]] where: - decimal record length, in frames (8-bit bytes) - # times this record length recurs. E - indicates error when reading this record (or the last of a series). is an optional # and specifies additional information, if any, as to nature of the error. The string "EOF" is a special that signals a tape mark. TAPE DIRECTORY EXAMPLE: Here is an example of a tape directory file for a small dump: ; DUMPER saveset of PS: ; Human-readable comment TF-Format: raw ; Required, and must be "raw" 0: 2590*8407 EOF ; First file 21774130: EOF ; Second file (empty == logical EOT) EOT: ; Physical EOT Note that comments can exist, that the data file format is RAW, and that it starts at frame 0 with a 2590-byte record which is repeated 8407 times, followed by an EOF tapemark, then another EOF tapemark. The emulator software generates the second EOF using a new line so that the tape location (21774130) can serve as a validity cross-check; this is a compromise between putting everything on a single line or putting each record on its own line, either of which is equally valid. "TPS" FILE FORMAT (.TAP, .TPS, .TPE) ==================================== This is the format used by Bob Supnik's SIMH emulators. It is almost the same as John Wilson's E11 format, which it was intended to resemble. A TPS file is assumed to consist of 8-bit bytes. Each record starts with a 4-byte header in little-endian order. The high bit of this 32-bit header is set to indicate an error of some kind (similar to the 'E' flag in a RAW-format control file) and the remaining 31 bits contain the record length N. This header is followed by N data bytes plus, if N is odd, a padding byte (of undefined value) at the end. Following this is a copy of the 4-byte record header. Tapemarks are represented by a single header with N=0. The reason for having the record length both before and after the record data is so tape motion in the reverse direction can be more easily simulated. The E11 (.TPE) format is identical except there are no padding bytes. "TPC" FILE FORMAT (.TPC) ======================== This format is not as flexible as the others but was apparently quite commonly used for DECUS tape images, and Tim Shoppa has a "couple thousand" TPC images stashed away. A TPC file is assumed to consist of 8-bit bytes. Each record starts with a 2-byte header in little-endian order, containing 16 bits of record length N. This header is followed by N data bytes plus, if N is odd, a padding byte (of undefined value) at the end. Unlike TPS format, there is no trailing header. Tapemarks are represented by a single header with N=0. Obviously it is difficult to use this format directly when reverse tape motion is desired; an internal representation must be built. TAPE GENERATION There are three principal ways to create virtual tapes: (1) Using the emulator to write tapes. (2) Copying from real PDP-10 tapes. (3) Creating synthesized tapes "by hand". TAPE GENERATION [1]: Using the emulator ======================================= Method [1] is described in the documentation for the emulator proper; see the section "Using Virtual Tapes" in USAGE.TXT. TAPE GENERATION [2]: Copying a real tape ======================================== This is likely to be the most common use of virtual tapes for a new installation. This is also the only way to move data in and out of a virtual system if there is no network support. The simplest method for copying a real tape is to use the TAPEDD utility provided with the emulator, as follows: % tapedd it=/dev/rmt0a otv=kosh This will copy the tape into a virtual tape named "kosh" with whatever extension is appropriate for the format. Note that the TAPEDD arguments have a form similar to that of the standard Unix DD utility, but are not the same. Invoking TAPEDD without arguments will cause it to list the possible options, most of which are never needed. If you want to copy a virtual tape back onto a physical tape, then the counterpart to the above command would be: % tapedd itv=kosh ot=/dev/rmt0a If for some reason TAPEDD is not available or you want to build a virtual tape that isn't a direct copy of an existing physical tape, you will need to get your hands dirty with method [3]. TAPEDD allows specifying the virtual tape format with a letter following the "v". For example: % tapedd it=/dev/rmt0a otvs=kosh will copy a tape into a TPS format file called "kosh.tps". TAPE GENERATION [3]: Creating a synthesized tape ================================================ One of the primary reasons for implementing virtual tapes as a RAW-format pair of files (data and directory) was to simplify the process of creating and managing these tapes by hand in an extremely portable way, without using any special software tools. To generate the tape directory file, all you need is a text editor. To generate the tape data file, normally it suffices to use the Unix utility "dd" or equivalent device-to-device copying program. You simply need to append every byte of tape data into a sequential binary file. For example, the tape data file corresponding to the sample tape directory above could have been read with a command like this: % dd if=/dev/rmt0a of=kosh.tap bs=126b 0+8407 records in 0+8407 records out Note that to use this technique properly you must have some prior knowledge of the record sizes on the tape. For one thing, the blocksize specified must be larger than the largest possible record size, to avoid record truncation (hence the bs=126b); but more importantly, without knowing the record size you don't know what to put in the tape directory (TDR) file. In this case, we know that TOPS-20 DUMPER records are <518 words * 5 bytes/word> = 2590 bytes and so the appropriate is 2590*8407. But if the records were of different lengths then DD provides no way for you to tell what these lengths were; this is one reason TAPEDD is easier. [Aside: TOPS-20 DUMPER could actually use a blocking factor of up to 15, so that records could be 15*518*5 = 38850 bytes long, but for virtual tapes the blocking is irrelevant and a factor of 1 always works for DUMPER.] To read multiple files from a tape using DD you must use a non-rewind device specification, such as "/dev/nrmt0a", and invoke DD once for each file. Remember to insert "EOF"s at appropriate places in the descriptor file. Although there is an UNIX utility called READ20 that can extract native files from TOPS-20 DUMPER format tapes (real or virtual), there is currently none for the inverse operation. In order to transport native files into the virtual system without a network, pretty much your only option is to synthesize a virtual tape where the contents are simple data files. WFCONV is a handy utility included with the emulator that will help prepare such files. It acts as a conversion filter, copying the standard input to the standard output and converting the data from one PDP-10 tape format to another. In order to fully understand why this is needed, you need to understand something about PDP-10 tape formats; for now, just note the following examples, and see the appendix for details. Example 1: transferring an ASCII text file: # Make NL -> CRLF and convert into core-dump format % wfconv -tc > temp.tap # Find size of file % ls -l temp.tap -rw-rw-r-- 1 klh klh10 18641 Apr 1 1994 temp.tap # Use size to compute number of 512-byte records # 18641/512 = 36, plus leftover record of 209 % cat > temp TF-Format: raw 0: 512*36 209 EOF EOF ; 36 records plus a short 37th EOT: ^D % ;; Then in TOPS-20 after mounting the tape "temp": ;; Note default format is CORE-DUMP, default reclen is 512. @copY (FROM) mta0: (TO) temp.txt MTA0: => TEMP.TXT.1 [OK] @vd temp.txt PS: TEMP.TXT.1;P777700 8 3729(36) 20-Apr-97 01:00:59 OPERATOR @ Example 2: transferring a binary file of 8-bit bytes (no conversion needed): % cp temp.dat temp.tap % ls -l temp.tap # Find size of file -rw-rw-r-- 1 klh klh10 18641 Feb 13 1994 temp.tap % cat > temp TF-Format: raw 0: 2560*7 721 EOF EOF EOT: ^D % ;; Then in TOPS-20 after mounting the tape "temp": @set taPE rECORD-LENGTH (TO) 2560 @set taPE fORMAT (TO) iNDUSTRY-COMPATIBLE @copY (FROM) mta0: (TO) temp.dat, @@byTE (SIZE) 8 @@ MTA0: => TEMP.DAT.1 [OK] @vd temp.dat PS: TEMP.DAT.1;P777700 10 18641(8) 19-Apr-97 23:57:46 OPERATOR @ Example 3: building a TOPS-20 installation tape If you already have a TOPS-20 system, especially one with local modifications (which pretty much includes everyone), you may wish to build your own installation tape. The following illustrates what the tape directory of the V7.0 TOPS-20 installation tape looks like. Note it does not have an initial bootstrap on it since MTBOOT is assumed to be already loaded in the PDP-10 memory. ; Tape directory for V7.0 TOPS-20 installation tape ; Bytes: 22519060, Records: 8704, Files(EOF marks): 7 TF-Format: raw 0: 2560*597 EOF ; MONITR.EXE 1528320: 2560*125 EOF ; EXEC.EXE 1848320: 2560*8 EOF ; DLUSER.EXE 1868800: 1270 EOF ; DLUSER data 1870070: 2560*36 EOF ; DUMPER.EXE 1962230: 2590*7937 EOF ; DUMPER savesets for , etc. 22519060: EOF EOT: All of these files must be in core-dump format. You can generate the tape either by: (1) creating it on a real TOPS system and then using it directly or using TAPEDD to copy it into a virtual tape. (2) copying the necessary files over, converting them into core-dump format, and concatenating them to form a virtual tape. See "klt20.txt" for more details on how such an installation tape is actually used. Quick lesson in tape formats: A PDP-10 word is 36 bits. The 8-bit bytes ("frames" on 9-track drives) that are now universal for tape don't fit exactly into 36 bits, which opens a Pandora's Box of packing options. The basic formats offered at one time or another on Digital systems are as follows: WFCONV mode TOPS-20 name ----------- ------------------ a ANSI-ASCII c CORE-DUMP h HIGH-DENSITY i INDUSTRY-COMPATIBLE s SIXBIT Only CORE-DUMP and INDUSTRY-COMPATIBLE are universally supported; the others are features of older or newer formatters. In general, the TOPS-20 monitor expects CORE-DUMP format so that should be your normal choice. Tape_Coredump (5 tape bytes per word) B0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 0 0 0 0 32 33 34 35 Tape_Indust (4 tape bytes per PARTIAL word) B0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 ; Bits 32-35 are unused! Tape_Sixbit (6 tape bytes per word) 0 0 B0 1 2 3 4 5 0 0 6 7 8 9 10 11 0 0 12 13 14 15 16 17 0 0 18 19 20 21 22 23 0 0 24 25 26 27 28 29 0 0 30 31 32 33 34 35 Tape_Ascii (5 tape bytes per word) 0 B0 1 2 3 4 5 6 0 7 8 9 10 11 12 13 0 14 15 16 17 18 19 20 0 21 22 23 24 25 26 27 35 28 29 30 31 32 33 34 Tape_Hidens (9 tape bytes per 2 words) B0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 B0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 WFCONV operation notes ====================== Invoking WFCONV without arguments will print out a summary similar to the following: Usage: wfconv -io outfile\n\ where 'i' and 'o' are chars specifying the input and output formats: c - Core-dump (std tape format, 4 8-bit, 1 4-bit bytes = 36 bits) h - High-density (FTP 36-bit image, 9 8-bit bytes = 72 bits) a,7 - Ansi-Ascii (4 7-bit, 1 8-bit byte = 36 bits) s,6 - Sixbit (6 6-bit bytes = 36 bits) u - Unixified (Alan Bawden format, various = 36 bits) t - Text-Newline (CRLF-NL conversion; 5 7-bit bytes = 35 bits ONLY) d - Debug (for output only) Note: EOF on input always causes zero-padding up to a PDP-10 word boundary. Typical uses of WFCONV would be as follows: 1. Examining a PDP-10 text file copied from a tape: # Convert from core-dump to ansi-ascii (preserve CRs) $ wfconv -ca < file.tap > file.asc or: # Convert from core-dump to text-newline (convert CRLFs to NLs) $ wfconv -ct < file.tap > file.txt 2. Preparing a text file for writing to tape: # Convert from ansi-ascii to core-dump (preserve NLs) $ wfconv -ac < file.asc > file.tap or: # Convert from text-newline to core-dump (convert NLs to CRLFs) $ wfconv -tc < file.txt > file.tap 3. Preparing a file for tape after transferring it from a real PDP-10 using FTP 36-bit Image mode: # Convert from high-density (FTP Image) to core-dump $ wfconv -hc < file.ftp > file.tap Note: Due to the fact that PDP-10 words are used as the canonical input conversion target (and output conversion source), input is always effectively rounded up to a word boundary. This can cause the addition of a few trailing zero bytes when converting from Text-Newline or Ansi-ASCII format, so WFCONV is not a general-purpose text-to-text conversion program. For that, one can use: # To insert CR (must use ^V or equiv to quote the ^M) $ sed 's/$/\^M/' temp.nls > temp.crlfs # To delete CR (must use ^V or equiv to quote the ^M) $ sed 's/\^M$//' temp.crlfs > temp.nls