GNU PSPP Developers Guide ¶

1.1 System File Record Structure ¶

System files are divided into records with the following format:

int32               type;
char                data[];

This header does not identify the length of the data or any information about what it contains, so the system file reader must understand the format of data based on type. However, records with type 7, called extension records, have a stricter format:

int32               type;
int32               subtype;
int32               size;
int32               count;
char                data[size * count];

int32 rec_type;

Record type. Always set to 7.

int32 subtype;

Record subtype. This value identifies a particular kind of extension record.

int32 size;

The size of each piece of data that follows the header, in bytes. Known extension records use 1, 4, or 8, for char, int32, and flt64 format data, respectively.

int32 count;

The number of pieces of data that follow the header.

char data[size * count];

Data, whose format and interpretation depend on the subtype.

An extension record contains exactly size * count bytes of data, which allows a reader that does not understand an extension record to skip it. Extension records provide only nonessential information, so this allows for files written by newer software to preserve backward compatibility with older or less capable readers.

Records in a system file must appear in the following order:

• File header record.
• Variable records.
• All pairs of value labels records and value label variables records, if present.
• Document record, if present.
• Extension (type 7) records, in ascending numerical order of their subtypes.

  System files written by SPSS include at most one of each kind of extension record. This is generally true of system files written by other software as well, with known exceptions noted below in the individual sections about each type of record.

• Dictionary termination record.
• Data record.

We advise authors of programs that read system files to tolerate format variations. Various kinds of misformatting and corruption have been observed in system files written by SPSS and other software alike. In particular, because extension records provide nonessential information, it is generally better to ignore an extension record entirely than to refuse to read a system file.

The following sections describe the known kinds of records.

1.2 File Header Record ¶

A system file begins with the file header, with the following format:

char                rec_type[4];
char                prod_name[60];
int32               layout_code;
int32               nominal_case_size;
int32               compression;
int32               weight_index;
int32               ncases;
flt64               bias;
char                creation_date[9];
char                creation_time[8];
char                file_label[64];
char                padding[3];

char rec_type[4];

Record type code, either ‘$FL2’ for system files with uncompressed data or data compressed with simple bytecode compression, or ‘$FL3’ for system files with ZLIB compressed data.

This is truly a character field that uses the character encoding as other strings. Thus, in a file with an ASCII-based character encoding this field contains 24 46 4c 32 or 24 46 4c 33, and in a file with an EBCDIC-based encoding this field contains 5b c6 d3 f2. (No EBCDIC-based ZLIB-compressed files have been observed.)

char prod_name[60];

Product identification string. This always begins with the characters ‘@(#) SPSS DATA FILE’. PSPP uses the remaining characters to give its version and the operating system name; for example, ‘GNU pspp 0.1.4 - sparc-sun-solaris2.5.2’. The string is truncated if it would be longer than 60 characters; otherwise it is padded on the right with spaces.

The product name field allow readers to behave differently based on quirks in the way that particular software writes system files. See Value Labels Records, for the detail of the quirk that the PSPP system file reader tolerates in files written by ReadStat, which has https://github.com/WizardMac/ReadStat in prod_name.

int32 layout_code;

Normally set to 2, although a few system files have been spotted in the wild with a value of 3 here. PSPP use this value to determine the file’s integer endianness (see System File Format).

int32 nominal_case_size;

Number of data elements per case. This is the number of variables, except that long string variables add extra data elements (one for every 8 characters after the first 8). However, string variables do not contribute to this value beyond the first 255 bytes. Further, some software always writes -1 or 0 in this field. In general, it is unsafe for systems reading system files to rely upon this value.

int32 compression;

Set to 0 if the data in the file is not compressed, 1 if the data is compressed with simple bytecode compression, 2 if the data is ZLIB compressed. This field has value 2 if and only if rec_type is ‘$FL3’.

int32 weight_index;

If one of the variables in the data set is used as a weighting variable, set to the dictionary index of that variable, plus 1 (see Dictionary Index). Otherwise, set to 0.

int32 ncases;

Set to the number of cases in the file if it is known, or -1 otherwise.

In the general case it is not possible to determine the number of cases that will be output to a system file at the time that the header is written. The way that this is dealt with is by writing the entire system file, including the header, then seeking back to the beginning of the file and writing just the ncases field. For files in which this is not valid, the seek operation fails. In this case, ncases remains -1.

flt64 bias;

Compression bias, ordinarily set to 100. Only integers between 1 - bias and 251 - bias can be compressed.

By assuming that its value is 100, PSPP uses bias to determine the file’s floating-point format and endianness (see System File Format). If the compression bias is not 100, PSPP cannot auto-detect the floating-point format and assumes that it is IEEE 754 format with the same endianness as the system file’s integers, which is correct for all known system files.

char creation_date[9];

Date of creation of the system file, in ‘dd mmm yy’ format, with the month as standard English abbreviations, using an initial capital letter and following with lowercase. If the date is not available then this field is arbitrarily set to ‘01 Jan 70’.

char creation_time[8];

Time of creation of the system file, in ‘hh:mm:ss’ format and using 24-hour time. If the time is not available then this field is arbitrarily set to ‘00:00:00’.

char file_label[64];

File label declared by the user, if any (see FILE LABEL in PSPP Users Guide). Padded on the right with spaces.

A product that identifies itself as VOXCO INTERVIEWER 4.3 uses CR-only line ends in this field, rather than the more usual LF-only or CR LF line ends.

char padding[3];

Ignored padding bytes to make the structure a multiple of 32 bits in length. Set to zeros.

1.3 Variable Record ¶

There must be one variable record for each numeric variable and each string variable with width 8 bytes or less. String variables wider than 8 bytes have one variable record for each 8 bytes, rounding up. The first variable record for a long string specifies the variable’s correct dictionary information. Subsequent variable records for a long string are filled with dummy information: a type of -1, no variable label or missing values, print and write formats that are ignored, and an empty string as name. A few system files have been encountered that include a variable label on dummy variable records, so readers should take care to parse dummy variable records in the same way as other variable records.

The dictionary index of a variable is a 1-based offset in the set of variable records, including dummy variable records for long string variables. The first variable record has a dictionary index of 1, the second has a dictionary index of 2, and so on.

The system file format does not directly support string variables wider than 255 bytes. Such very long string variables are represented by a number of narrower string variables. See Very Long String Record, for details.

A system file should contain at least one variable and thus at least one variable record, but system files have been observed in the wild without any variables (thus, no data either).

int32               rec_type;
int32               type;
int32               has_var_label;
int32               n_missing_values;
int32               print;
int32               write;
char                name[8];

/* Present only if has_var_label is 1. */
int32               label_len;
char                label[];

/* Present only if n_missing_values is nonzero. */
flt64               missing_values[];

int32 rec_type;

Record type code. Always set to 2.

int32 type;

Variable type code. Set to 0 for a numeric variable. For a short string variable or the first part of a long string variable, this is set to the width of the string. For the second and subsequent parts of a long string variable, set to -1, and the remaining fields in the structure are ignored.

int32 has_var_label;

If this variable has a variable label, set to 1; otherwise, set to 0.

int32 n_missing_values;

If the variable has no missing values, set to 0. If the variable has one, two, or three discrete missing values, set to 1, 2, or 3, respectively. If the variable has a range for missing variables, set to -2; if the variable has a range for missing variables plus a single discrete value, set to -3.

A long string variable always has the value 0 here. A separate record indicates missing values for long string variables (see Long String Missing Values Record).

int32 print;

Print format for this variable. See below.

int32 write;

Write format for this variable. See below.

char name[8];

Variable name. The variable name must begin with a capital letter or the at-sign (‘@’). Subsequent characters may also be digits, octothorpes (‘#’), dollar signs (‘$’), underscores (‘_’), or full stops (‘.’). The variable name is padded on the right with spaces.

The ‘name’ fields should be unique within a system file. System files written by SPSS that contain very long string variables with similar names sometimes contain duplicate names that are later eliminated by resolving the very long string names (see Very Long String Record). PSPP handles duplicates by assigning them new, unique names.

int32 label_len;

This field is present only if has_var_label is set to 1. It is set to the length, in characters, of the variable label. The documented maximum length varies from 120 to 255 based on SPSS version, but some files have been seen with longer labels. PSPP accepts labels of any length.

char label[];

This field is present only if has_var_label is set to 1. It has length label_len, rounded up to the nearest multiple of 32 bits. The first label_len characters are the variable’s variable label.

flt64 missing_values[];

This field is present only if n_missing_values is nonzero. It has the same number of 8-byte elements as the absolute value of n_missing_values. Each element is interpreted as a number for numeric variables (with HIGHEST and LOWEST indicated as described in the chapter introduction). For string variables of width less than 8 bytes, elements are right-padded with spaces; for string variables wider than 8 bytes, only the first 8 bytes of each missing value are specified, with the remainder implicitly all spaces.

For discrete missing values, each element represents one missing value. When a range is present, the first element denotes the minimum value in the range, and the second element denotes the maximum value in the range. When a range plus a value are present, the third element denotes the additional discrete missing value.

The print and write members of sysfile_variable are output formats coded into int32 types. The least-significant byte of the int32 represents the number of decimal places, and the next two bytes in order of increasing significance represent field width and format type, respectively. The most-significant byte is not used and should be set to zero.

Format types are defined as follows:

Value	Meaning
0	Not used.
1	A
2	AHEX
3	COMMA
4	DOLLAR
5	F
6	IB
7	PIBHEX
8	P
9	PIB
10	PK
11	RB
12	RBHEX
13	Not used.
14	Not used.
15	Z
16	N
17	E
18	Not used.
19	Not used.
20	DATE
21	TIME
22	DATETIME
23	ADATE
24	JDATE
25	DTIME
26	WKDAY
27	MONTH
28	MOYR
29	QYR
30	WKYR
31	PCT
32	DOT
33	CCA
34	CCB
35	CCC
36	CCD
37	CCE
38	EDATE
39	SDATE
40	MTIME
41	YMDHMS

A few system files have been observed in the wild with invalid write fields, in particular with value 0. Readers should probably treat invalid print or write fields as some default format.

1.4 Value Labels Records ¶

The value label records documented in this section are used for numeric and short string variables only. Long string variables may have value labels, but their value labels are recorded using a different record type (see Long String Value Labels Record).

ReadStat (see File Header Record) writes value labels that label a single value more than once. In more detail, it emits value labels whose values are longer than string variables’ widths, that are identical in the actual width of the variable, e.g. labels for values ABC123 and ABC456 for a string variable with width 3. For files written by this software, PSPP ignores such labels.

The value label record has the following format:

int32               rec_type;
int32               label_count;

/* Repeated n_label times. */
char                value[8];
char                label_len;
char                label[];

int32 rec_type;

Record type. Always set to 3.

int32 label_count;

Number of value labels present in this record.

The remaining fields are repeated count times. Each repetition specifies one value label.

char value[8];

A numeric value or a short string value padded as necessary to 8 bytes in length. Its type and width cannot be determined until the following value label variables record (see below) is read.

char label_len;

The label’s length, in bytes. The documented maximum length varies from 60 to 120 based on SPSS version. PSPP supports value labels up to 255 bytes long.

char label[];

label_len bytes of the actual label, followed by up to 7 bytes of padding to bring label and label_len together to a multiple of 8 bytes in length.

The value label record is always immediately followed by a value label variables record with the following format:

int32               rec_type;
int32               var_count;
int32               vars[];

int32 rec_type;

Record type. Always set to 4.

int32 var_count;

Number of variables that the associated value labels from the value label record are to be applied.

int32 vars[];

A list of 1-based dictionary indexes of variables to which to apply the value labels (see Dictionary Index). There are var_count elements.

String variables wider than 8 bytes may not be specified in this list.

1.5 Document Record ¶

The document record, if present, has the following format:

int32               rec_type;
int32               n_lines;
char                lines[][80];

int32 rec_type;

Record type. Always set to 6.

int32 n_lines;

Number of lines of documents present. This should be greater than zero, but ReadStats writes system files with zero n_lines.

char lines[][80];

Document lines. The number of elements is defined by n_lines. Lines shorter than 80 characters are padded on the right with spaces.

1.6 Machine Integer Info Record ¶

The integer info record, if present, has the following format:

/* Header. */
int32               rec_type;
int32               subtype;
int32               size;
int32               count;

/* Data. */
int32               version_major;
int32               version_minor;
int32               version_revision;
int32               machine_code;
int32               floating_point_rep;
int32               compression_code;
int32               endianness;
int32               character_code;

int32 rec_type;

Record type. Always set to 7.

int32 subtype;

Record subtype. Always set to 3.

int32 size;

Size of each piece of data in the data part, in bytes. Always set to 4.

int32 count;

Number of pieces of data in the data part. Always set to 8.

int32 version_major;

PSPP major version number. In version x.y.z, this is x.

int32 version_minor;

PSPP minor version number. In version x.y.z, this is y.

int32 version_revision;

PSPP version revision number. In version x.y.z, this is z.

int32 machine_code;

Machine code. PSPP always set this field to value to -1, but other values may appear.

int32 floating_point_rep;

Floating point representation code. For IEEE 754 systems this is 1. IBM 370 sets this to 2, and DEC VAX E to 3.

int32 compression_code;

Compression code. Always set to 1, regardless of whether or how the file is compressed.

int32 endianness;

Machine endianness. 1 indicates big-endian, 2 indicates little-endian.

int32 character_code;

Character code. The following values have been actually observed in system files:

1

EBCDIC.

2

7-bit ASCII.

1250

The windows-1250 code page for Central European and Eastern European languages.

1252

The windows-1252 code page for Western European languages.

28591

ISO 8859-1.

65001

UTF-8.

The following additional values are known to be defined:

3

8-bit “ASCII”.

4

DEC Kanji.

Other Windows code page numbers are known to be generally valid.

Old versions of SPSS for Unix and Windows always wrote value 2 in this field, regardless of the encoding in use. Newer versions also write the character encoding as a string (see Character Encoding Record).

1.7 Machine Floating-Point Info Record ¶

The floating-point info record, if present, has the following format:

/* Header. */
int32               rec_type;
int32               subtype;
int32               size;
int32               count;

/* Data. */
flt64               sysmis;
flt64               highest;
flt64               lowest;

int32 rec_type;

Record type. Always set to 7.

int32 subtype;

Record subtype. Always set to 4.

int32 size;

Size of each piece of data in the data part, in bytes. Always set to 8.

int32 count;

Number of pieces of data in the data part. Always set to 3.

flt64 sysmis;

flt64 highest;

flt64 lowest;

The system missing value, the value used for HIGHEST in missing values, and the value used for LOWEST in missing values, respectively. See System File Format, for more information.

The SPSSWriter library in PHP, which identifies itself as FOM SPSS 1.0.0 in the file header record prod_name field, writes unexpected values to these fields, but it uses the same values consistently throughout the rest of the file.

1.8 Multiple Response Sets Records ¶

The system file format has two different types of records that represent multiple response sets (see MRSETS in PSPP Users Guide). The first type of record describes multiple response sets that can be understood by SPSS before version 14. The second type of record, with a closely related format, is used for multiple dichotomy sets that use the CATEGORYLABELS=COUNTEDVALUES feature added in version 14.

/* Header. */
int32               rec_type;
int32               subtype;
int32               size;
int32               count;

/* Exactly count bytes of data. */
char                mrsets[];

int32 rec_type;

Record type. Always set to 7.

int32 subtype;

Record subtype. Set to 7 for records that describe multiple response sets understood by SPSS before version 14, or to 19 for records that describe dichotomy sets that use the CATEGORYLABELS=COUNTEDVALUES feature added in version 14.

int32 size;

The size of each element in the mrsets member. Always set to 1.

int32 count;

The total number of bytes in mrsets.

char mrsets[];

Zero or more line feeds (byte 0x0a), followed by a series of multiple response sets, each of which consists of the following:

• The set’s name (an identifier that begins with ‘$’), in mixed upper and lower case.
• An equals sign (‘=’).
• ‘C’ for a multiple category set, ‘D’ for a multiple dichotomy set with CATEGORYLABELS=VARLABELS, or ‘E’ for a multiple dichotomy set with CATEGORYLABELS=COUNTEDVALUES.
• For a multiple dichotomy set with CATEGORYLABELS=COUNTEDVALUES, a space, followed by a number expressed as decimal digits, followed by a space. If LABELSOURCE=VARLABEL was specified on MRSETS, then the number is 11; otherwise it is 1.¹
• For either kind of multiple dichotomy set, the counted value, as a positive integer count specified as decimal digits, followed by a space, followed by as many string bytes as specified in the count. If the set contains numeric variables, the string consists of the counted integer value expressed as decimal digits. If the set contains string variables, the string contains the counted string value. Either way, the string may be padded on the right with spaces (older versions of SPSS seem to always pad to a width of 8 bytes; newer versions don’t).
• A space.
• The multiple response set’s label, using the same format as for the counted value for multiple dichotomy sets. A string of length 0 means that the set does not have a label. A string of length 0 is also written if LABELSOURCE=VARLABEL was specified.
• A space.
• The short names of the variables in the set, converted to lowercase, each separated from the previous by a single space.

  Even though a multiple response set must have at least two variables, some system files contain multiple response sets with no variables or one variable. The source and meaning of these multiple response sets is unknown. (Perhaps they arise from creating a multiple response set then deleting all the variables that it contains?)

• One line feed (byte 0x0a). Sometimes multiple, even hundreds, of line feeds are present.

Example: Given appropriate variable definitions, consider the following MRSETS command:

MRSETS /MCGROUP NAME=$a LABEL='my mcgroup' VARIABLES=a b c
       /MDGROUP NAME=$b VARIABLES=g e f d VALUE=55
       /MDGROUP NAME=$c LABEL='mdgroup #2' VARIABLES=h i j VALUE='Yes'
       /MDGROUP NAME=$d LABEL='third mdgroup' CATEGORYLABELS=COUNTEDVALUES
        VARIABLES=k l m VALUE=34
       /MDGROUP NAME=$e CATEGORYLABELS=COUNTEDVALUES LABELSOURCE=VARLABEL
        VARIABLES=n o p VALUE='choice'.

The above would generate the following multiple response set record of subtype 7:

$a=C 10 my mcgroup a b c
$b=D2 55 0  g e f d
$c=D3 Yes 10 mdgroup #2 h i j

It would also generate the following multiple response set record with subtype 19:

$d=E 1 2 34 13 third mdgroup k l m
$e=E 11 6 choice 0  n o p

1.9 Extra Product Info Record ¶

This optional record appears to contain a text string that describes the program that wrote the file and the source of the data. (This is redundant with the file label and product info found in the file header record.)

/* Header. */
int32               rec_type;
int32               subtype;
int32               size;
int32               count;

/* Exactly count bytes of data. */
char                info[];

int32 rec_type;

Record type. Always set to 7.

int32 subtype;

Record subtype. Always set to 10.

int32 size;

The size of each element in the info member. Always set to 1.

int32 count;

The total number of bytes in info.

char info[];

A text string. A product that identifies itself as VOXCO INTERVIEWER 4.3 uses CR-only line ends in this field, rather than the more usual LF-only or CR LF line ends.

1.10 Variable Display Parameter Record ¶

The variable display parameter record, if present, has the following format:

/* Header. */
int32               rec_type;
int32               subtype;
int32               size;
int32               count;

/* Repeated count times. */
int32               measure;
int32               width;           /* Not always present. */
int32               alignment;

int32 rec_type;

Record type. Always set to 7.

int32 subtype;

Record subtype. Always set to 11.

int32 size;

The size of int32. Always set to 4.

int32 count;

The number of sets of variable display parameters (ordinarily the number of variables in the dictionary), times 2 or 3.

The remaining members are repeated count times, in the same order as the variable records. No element corresponds to variable records that continue long string variables. The meanings of these members are as follows:

int32 measure;

The measurement level of the variable:

0

Unknown

1

Nominal

2

Ordinal

3

Scale

An “unknown” measure of 0 means that the variable was created in some way that doesn’t make the measurement level clear, e.g. with a COMPUTE transformation. PSPP sets the measurement level the first time it reads the data using the rules documented in Measurement Level in PSPP Users Guide, so this should rarely appear.

int32 width;

The width of the display column for the variable in characters.

This field is present if count is 3 times the number of variables in the dictionary. It is omitted if count is 2 times the number of variables.

int32 alignment;

The alignment of the variable for display purposes:

0

Left aligned

1

Right aligned

2

Centre aligned

1.11 Variable Sets Record ¶

The SPSS GUI offers users the ability to arrange variables in sets. Users may enable and disable sets individually, and the data editor and analysis dialog boxes only show enabled sets. Syntax does not use variable sets.

The variable sets record, if present, has the following format:

/* Header. */
int32               rec_type;
int32               subtype;
int32               size;
int32               count;

/* Exactly count bytes of text. */
char                text[];

int32 rec_type;

Record type. Always set to 7.

int32 subtype;

Record subtype. Always set to 5.

int32 size;

Always set to 1.

int32 count;

The total number of bytes in text.

char text[];

The variable sets, in a text-based format.

Each variable set occupies one line of text, each of which ends with a line feed (byte 0x0a), optionally preceded by a carriage return (byte 0x0d).

Each line begins with the name of the variable set, followed by an equals sign (‘=’) and a space (byte 0x20), followed by the long variable names of the members of the set, separated by spaces. A variable set may be empty, in which case the equals sign and the space following it are still present.

1.12 Long Variable Names Record ¶

If present, the long variable names record has the following format:

/* Header. */
int32               rec_type;
int32               subtype;
int32               size;
int32               count;

/* Exactly count bytes of data. */
char                var_name_pairs[];

int32 rec_type;

Record type. Always set to 7.

int32 subtype;

Record subtype. Always set to 13.

int32 size;

The size of each element in the var_name_pairs member. Always set to 1.

int32 count;

The total number of bytes in var_name_pairs.

char var_name_pairs[];

A list of key–value tuples, where key is the name of a variable, and value is its long variable name. The key field is at most 8 bytes long and must match the name of a variable which appears in the variable record (see Variable Record). The value field is at most 64 bytes long. The key and value fields are separated by a ‘=’ byte. Each tuple is separated by a byte whose value is 09. There is no trailing separator following the last tuple. The total length is count bytes.

1.13 Very Long String Record ¶

Old versions of SPSS limited string variables to a width of 255 bytes. For backward compatibility with these older versions, the system file format represents a string longer than 255 bytes, called a very long string, as a collection of strings no longer than 255 bytes each. The strings concatenated to make a very long string are called its segments; for consistency, variables other than very long strings are considered to have a single segment.

A very long string with a width of w has n = (w + 251) / 252 segments, that is, one segment for every 252 bytes of width, rounding up. It would be logical, then, for each of the segments except the last to have a width of 252 and the last segment to have the remainder, but this is not the case. In fact, each segment except the last has a width of 255 bytes. The last segment has width w - (n - 1) * 252; some versions of SPSS make it slightly wider, but not wide enough to make the last segment require another 8 bytes of data.

Data is packed tightly into segments of a very long string, 255 bytes per segment. Because 255 bytes of segment data are allocated for every 252 bytes of the very long string’s width (approximately), some unused space is left over at the end of the allocated segments. Data in unused space is ignored.

Example: Consider a very long string of width 20,000. Such a very long string has 20,000 / 252 = 80 (rounding up) segments. The first 79 segments have width 255; the last segment has width 20,000 - 79 * 252 = 92 or slightly wider (up to 96 bytes, the next multiple of 8). The very long string’s data is actually stored in the 19,890 bytes in the first 78 segments, plus the first 110 bytes of the 79th segment (19,890 + 110 = 20,000). The remaining 145 bytes of the 79th segment and all 92 bytes of the 80th segment are unused.

The very long string record explains how to stitch together segments to obtain very long string data. For each of the very long string variables in the dictionary, it specifies the name of its first segment’s variable and the very long string variable’s actual width. The remaining segments immediately follow the named variable in the system file’s dictionary.

The very long string record, which is present only if the system file contains very long string variables, has the following format:

/* Header. */
int32               rec_type;
int32               subtype;
int32               size;
int32               count;

/* Exactly count bytes of data. */
char                string_lengths[];

int32 rec_type;

Record type. Always set to 7.

int32 subtype;

Record subtype. Always set to 14.

int32 size;

The size of each element in the string_lengths member. Always set to 1.

int32 count;

The total number of bytes in string_lengths.

char string_lengths[];

A list of key–value tuples, where key is the name of a variable, and value is its length. The key field is at most 8 bytes long and must match the name of a variable which appears in the variable record (see Variable Record). The value field is exactly 5 bytes long. It is a zero-padded, ASCII-encoded string that is the length of the variable. The key and value fields are separated by a ‘=’ byte. Tuples are delimited by a two-byte sequence {00, 09}. After the last tuple, there may be a single byte 00, or {00, 09}. The total length is count bytes.

1.14 Character Encoding Record ¶

This record, if present, indicates the character encoding for string data, long variable names, variable labels, value labels and other strings in the file.

/* Header. */
int32               rec_type;
int32               subtype;
int32               size;
int32               count;

/* Exactly count bytes of data. */
char                encoding[];

int32 rec_type;

Record type. Always set to 7.

int32 subtype;

Record subtype. Always set to 20.

int32 size;

The size of each element in the encoding member. Always set to 1.

int32 count;

The total number of bytes in encoding.

char encoding[];

The name of the character encoding. Normally this will be an official IANA character set name or alias. See http://www.iana.org/assignments/character-sets. Character set names are not case-sensitive, but SPSS appears to write them in all-uppercase.

This record is not present in files generated by older software. See also the character_code field in the machine integer info record (see character-code).

When the character encoding record and the machine integer info record are both present, all system files observed in practice indicate the same character encoding, e.g. 1252 as character_code and windows-1252 as encoding, 65001 and UTF-8, etc.

If, for testing purposes, a file is crafted with different character_code and encoding, it seems that character_code controls the encoding for all strings in the system file before the dictionary termination record, including strings in data (e.g. string missing values), and encoding controls the encoding for strings following the dictionary termination record.

1.15 Long String Value Labels Record ¶

This record, if present, specifies value labels for long string variables.

/* Header. */
int32               rec_type;
int32               subtype;
int32               size;
int32               count;

/* Repeated up to exactly count bytes. */
int32               var_name_len;
char                var_name[];
int32               var_width;
int32               n_labels;
long_string_label   labels[];

int32 rec_type;

Record type. Always set to 7.

int32 subtype;

Record subtype. Always set to 21.

int32 size;

Always set to 1.

int32 count;

The number of bytes following the header until the next header.

int32 var_name_len;

char var_name[];

The number of bytes in the name of the variable that has long string value labels, plus the variable name itself, which consists of exactly var_name_len bytes. The variable name is not padded to any particular boundary, nor is it null-terminated.

int32 var_width;

The width of the variable, in bytes, which will be between 9 and 32767.

int32 n_labels;

long_string_label labels[];

The long string labels themselves. The labels array contains exactly n_labels elements, each of which has the following substructure:

int32               value_len;
char                value[];
int32               label_len;
char                label[];

int32 value_len;

char value[];

The string value being labeled. value_len is the number of bytes in value; it is equal to var_width. The value array is not padded or null-terminated.

int32 label_len;

char label[];

The label for the string value. label_len, which must be between 0 and 120, is the number of bytes in label. The label array is not padded or null-terminated.

1.16 Long String Missing Values Record ¶

This record, if present, specifies missing values for long string variables.

/* Header. */
int32               rec_type;
int32               subtype;
int32               size;
int32               count;

/* Repeated up to exactly count bytes. */
int32               var_name_len;
char                var_name[];
char                n_missing_values;
int32               value_len;
char                values[values_len * n_missing_values];

int32 rec_type;

Record type. Always set to 7.

int32 subtype;

Record subtype. Always set to 22.

int32 size;

Always set to 1.

int32 count;

The number of bytes following the header until the next header.

int32 var_name_len;

char var_name[];

The number of bytes in the name of the long string variable that has missing values, plus the variable name itself, which consists of exactly var_name_len bytes. The variable name is not padded to any particular boundary, nor is it null-terminated.

char n_missing_values;

The number of missing values, either 1, 2, or 3. (This is, unusually, a single byte instead of a 32-bit number.)

int32 value_len;

The length of each missing value string, in bytes. This value should be 8, because long string variables are at least 8 bytes wide (by definition), only the first 8 bytes of a long string variable’s missing values are allowed to be non-spaces, and any spaces within the first 8 bytes are included in the missing value here.

char values[values_len * n_missing_values]

The missing values themselves, without any padding or null terminators.

An earlier version of this document stated that value_len was repeated before each of the missing values, so that there was an extra int32 value of 8 before each missing value after the first. Old versions of PSPP wrote data files in this format. Readers can tolerate this mistake, if they wish, by noticing and skipping the extra int32 values, which wouldn’t ordinarily occur in strings.

1.17 Data File and Variable Attributes Records ¶

The data file and variable attributes records represent custom attributes for the system file or for individual variables in the system file, as defined on the DATAFILE ATTRIBUTE (see DATAFILE ATTRIBUTE in PSPP Users Guide) and VARIABLE ATTRIBUTE commands (see VARIABLE ATTRIBUTE in PSPP Users Guide), respectively.

/* Header. */
int32               rec_type;
int32               subtype;
int32               size;
int32               count;

/* Exactly count bytes of data. */
char                attributes[];

int32 rec_type;

Record type. Always set to 7.

int32 subtype;

Record subtype. Always set to 17 for a data file attribute record or to 18 for a variable attributes record.

int32 size;

The size of each element in the attributes member. Always set to 1.

int32 count;

The total number of bytes in attributes.

char attributes[];

The attributes, in a text-based format.

In record subtype 17, this field contains a single attribute set. An attribute set is a sequence of one or more attributes concatenated together. Each attribute consists of a name, which has the same syntax as a variable name, followed by, inside parentheses, a sequence of one or more values. Each value consists of a string enclosed in single quotes (') followed by a line feed (byte 0x0a). A value may contain single quote characters, which are not themselves escaped or quoted or required to be present in pairs. There is no apparent way to embed a line feed in a value. There is no distinction between an attribute with a single value and an attribute array with one element.

In record subtype 18, this field contains a sequence of one or more variable attribute sets. If more than one variable attribute set is present, each one after the first is delimited from the previous by /. Each variable attribute set consists of a long variable name, followed by :, followed by an attribute set with the same syntax as on record subtype 17.

System files written by Stata 14.1/-savespss- 1.77 by S.Radyakin may include multiple records with subtype 18, one per variable that has variable attributes.

The total length is count bytes.

Example ¶

A system file produced with the following VARIABLE ATTRIBUTE commands in effect:

VARIABLE ATTRIBUTE VARIABLES=dummy ATTRIBUTE=fred[1]('23') fred[2]('34').
VARIABLE ATTRIBUTE VARIABLES=dummy ATTRIBUTE=bert('123').

will contain a variable attribute record with the following contents:

0000  07 00 00 00 12 00 00 00  01 00 00 00 22 00 00 00  |............"...|
0010  64 75 6d 6d 79 3a 66 72  65 64 28 27 32 33 27 0a  |dummy:fred('23'.|
0020  27 33 34 27 0a 29 62 65  72 74 28 27 31 32 33 27  |'34'.)bert('123'|
0030  0a 29                                             |.)              |

• Variable Roles

1.18 Extended Number of Cases Record ¶

The file header record expresses the number of cases in the system file as an int32 (see File Header Record). This record allows the number of cases in the system file to be expressed as a 64-bit number.

int32               rec_type;
int32               subtype;
int32               size;
int32               count;
int64               unknown;
int64               ncases64;

int32 rec_type;

Record type. Always set to 7.

int32 subtype;

Record subtype. Always set to 16.

int32 size;

Size of each element. Always set to 8.

int32 count;

Number of pieces of data in the data part. Alway set to 2.

int64 unknown;

Meaning unknown. Always set to 1.

int64 ncases64;

Number of cases in the file as a 64-bit integer. Presumably this could be -1 to indicate that the number of cases is unknown, for the same reason as ncases in the file header record, but this has not been observed in the wild.

1.19 Other Informational Records ¶

This chapter documents many specific types of extension records are documented here, but others are known to exist. PSPP ignores unknown extension records when reading system files.

The following extension record subtypes have also been observed, with the following believed meanings:

6

Date info, probably related to USE (according to Aapi Hämäläinen).

12

A UUID in the format described in RFC 4122. Only two examples observed, both written by SPSS 13, and in each case the UUID contained both upper and lower case.

24

XML that describes how data in the file should be displayed on-screen.

1.20 Dictionary Termination Record ¶

The dictionary termination record separates all other records from the data records.

int32               rec_type;
int32               filler;

int32 rec_type;

Record type. Always set to 999.

int32 filler;

Ignored padding. Should be set to 0.

1.21 Data Record ¶

The data record must follow all other records in the system file. Every system file must have a data record that specifies data for at least one case. The format of the data record varies depending on the value of compression in the file header record:

0: no compression

Data is arranged as a series of 8-byte elements. Each element corresponds to the variable declared in the respective variable record (see Variable Record). Numeric values are given in flt64 format; string values are literal characters string, padded on the right when necessary to fill out 8-byte units.

1: bytecode compression

The first 8 bytes of the data record is divided into a series of 1-byte command codes. These codes have meanings as described below:

0

Ignored. If the program writing the system file accumulates compressed data in blocks of fixed length, 0 bytes can be used to pad out extra bytes remaining at the end of a fixed-size block.

1 through 251

A number with value code - bias, where code is the value of the compression code and bias is the variable bias from the file header. For example, code 105 with bias 100.0 (the normal value) indicates a numeric variable of value 5.

A code of 0 (after subtracting the bias) in a string field encodes null bytes. This is unusual, since a string field normally encodes text data, but it exists in real system files.

252

End of file. This code may or may not appear at the end of the data stream. PSPP always outputs this code but its use is not required.

253

A numeric or string value that is not compressible. The value is stored in the 8 bytes following the current block of command bytes. If this value appears twice in a block of command bytes, then it indicates the second group of 8 bytes following the command bytes, and so on.

254

An 8-byte string value that is all spaces.

255

The system-missing value.

The end of the 8-byte group of bytecodes is followed by any 8-byte blocks of non-compressible values indicated by code 253. After that follows another 8-byte group of bytecodes, then those bytecodes’ non-compressible values. The pattern repeats to the end of the file or a code with value 252.

2: ZLIB compression

The data record consists of the following, in order:

• ZLIB data header, 24 bytes long.
• One or more variable-length blocks of ZLIB compressed data.
• ZLIB data trailer, with a 24-byte fixed header plus an additional 24 bytes for each preceding ZLIB compressed data block.

The ZLIB data header has the following format:

int64               zheader_ofs;
int64               ztrailer_ofs;
int64               ztrailer_len;

int64 zheader_ofs;

The offset, in bytes, of the beginning of this structure within the system file.

int64 ztrailer_ofs;

The offset, in bytes, of the first byte of the ZLIB data trailer.

int64 ztrailer_len;

The number of bytes in the ZLIB data trailer. This and the previous field sum to the size of the system file in bytes.

The data header is followed by (ztrailer_len - 24) / 24 ZLIB compressed data blocks. Each ZLIB compressed data block begins with a ZLIB header as specified in RFC 1950, e.g. hex bytes 78 01 (the only header yet observed in practice). Each block decompresses to a fixed number of bytes (in practice only 0x3ff000-byte blocks have been observed), except that the last block of data may be shorter. The last ZLIB compressed data block gends just before offset ztrailer_ofs.

The result of ZLIB decompression is bytecode compressed data as described above for compression format 1.

The ZLIB data trailer begins with the following 24-byte fixed header:

int64               bias;
int64               zero;
int32               block_size;
int32               n_blocks;

int64 int_bias;

The compression bias as a negative integer, e.g. if bias in the file header record is 100.0, then int_bias is −100 (this is the only value yet observed in practice).

int64 zero;

Always observed to be zero.

int32 block_size;

The number of bytes in each ZLIB compressed data block, except possibly the last, following decompression. Only 0x3ff000 has been observed so far.

int32 n_blocks;

The number of ZLIB compressed data blocks, always exactly (ztrailer_len - 24) / 24.

The fixed header is followed by n_blocks 24-byte ZLIB data block descriptors, each of which describes the compressed data block corresponding to its offset. Each block descriptor has the following format:

int64               uncompressed_ofs;
int64               compressed_ofs;
int32               uncompressed_size;
int32               compressed_size;

int64 uncompressed_ofs;

The offset, in bytes, that this block of data would have in a similar system file that uses compression format 1. This is zheader_ofs in the first block descriptor, and in each succeeding block descriptor it is the sum of the previous desciptor’s uncompressed_ofs and uncompressed_size.

int64 compressed_ofs;

The offset, in bytes, of the actual beginning of this compressed data block. This is zheader_ofs + 24 in the first block descriptor, and in each succeeding block descriptor it is the sum of the previous descriptor’s compressed_ofs and compressed_size. The final block descriptor’s compressed_ofs and compressed_size sum to ztrailer_ofs.

int32 uncompressed_size;

The number of bytes in this data block, after decompression. This is block_size in every data block except the last, which may be smaller.

int32 compressed_size;

The number of bytes in this data block, as stored compressed in this system file.

2.1 Structure Member Format ¶

A structure member lays out the high-level structure for a group of output items such as heading, tables, and charts. Structure members do not include the details of tables and charts but instead refer to them by their member names.

Structure members’ XML files claim conformance with a collection of XML Schemas. These schemas are distributed, under a nonfree license, with SPSS binaries. Fortunately, the schemas are not necessary to understand the structure members. The schemas can even be deceptive because they document elements and attributes that are not in the corpus and do not document elements and attributes that are commonly found in the corpus.

Structure members use a different XML namespace for each schema, but these namespaces are not entirely consistent. In some SPV files, for example, the viewer-tree schema is associated with namespace ‘http://xml.spss.com/spss/viewer-tree’ and in others with ‘http://xml.spss.com/spss/viewer/viewer-tree’ (note the additional viewer/). Under either name, the schema URIs are not resolvable to obtain the schemas themselves.

One may ignore all of the above in interpreting a structure member. The actual XML has a simple and straightforward form that does not require a reader to take schemas or namespaces into account. A structure member’s root is heading element, which contains heading or container elements (or a mix), forming a tree. In turn, container holds a label and one more child, usually text or table.

The following sections document the elements found in structure members in a context-free grammar-like fashion. Consider the following example, which specifies the attributes and content for the container element:

container
   :visibility=(visible | hidden)
   :page-break-before=(always)?
   :text-align=(left | center)?
   :width=dimension
=> label (table | container_text | graph | model | object | image | tree)

Each attribute specification begins with ‘:’ followed by the attribute’s name. If the attribute’s value has an easily specified form, then ‘=’ and its description follows the name. Finally, if the attribute is optional, the specification ends with ‘?’. The following value specifications are defined:

(a | b | …)

One of the listed literal strings. If only one string is listed, it is the only acceptable value. If OTHER is listed, then any string not explicitly listed is also accepted.

bool

Either true or false.

dimension

A floating-point number followed by a unit, e.g. 10pt. Units in the corpus include in (inch), pt (points, 72/inch), px (“device-independent pixels”, 96/inch), and cm. If the unit is omitted then points should be assumed. The number and unit may be separated by white space.

The corpus also includes localized names for units. A reader must understand these to properly interpret the dimension:

inch

인치, pol., cala, cali

point

пт

centimeter

см

real

A floating-point number.

int

An integer.

color

A color in one of the forms #rrggbb or rrggbb, or the string transparent, or one of the standard Web color names.

ref

ref element

ref(elem1 | elem2 | …)

The name from the id attribute in some element. If one or more elements are named, the name must refer to one of those elements, otherwise any element is acceptable.

All elements have an optional id attribute. If present, its value must be unique. In practice many elements are assigned id attributes that are never referenced.

The content specification for an element supports the following syntax:

element

An element.

a b

a followed by b.

a | b | c

One of a or b or c.

a?

Zero or one instances of a.

a*

Zero or more instances of a.

b+

One or more instances of a.

(subexpression)

Grouping for a subexpression.

EMPTY

No content.

TEXT

Text and CDATA.

Element and attribute names are sometimes suffixed by another name in square brackets to distinguish different uses of the same name. For example, structure XML has two text elements, one inside container, the other inside pageParagraph. The former is defined as text[container_text] and referenced as container_text, the latter defined as text[pageParagraph_text] and referenced as pageParagraph_text.

This language is used in the PSPP source code for parsing structure and detail XML members. Refer to src/output/spv/structure-xml.grammar and src/output/spv/detail-xml.grammar for the full grammars.

The following example shows the contents of a typical structure member for a DESCRIPTIVES procedure. A real structure member is not indented. This example also omits most attributes, all XML namespace information, and the CSS from the embedded HTML:

<?xml version="1.0" encoding="utf-8"?>
<heading>
  <label>Output</label>
  <heading commandName="Descriptives">
    <label>Descriptives</label>
    <container>
      <label>Title</label>
      <text commandName="Descriptives" type="title">
        <html lang="en">
<![CDATA[<head><style type="text/css">...</style></head><BR>Descriptives]]>
        </html>
      </text>
    </container>
    <container visibility="hidden">
      <label>Notes</label>
      <table commandName="Descriptives" subType="Notes" type="note">
        <tableStructure>
          <dataPath>00000000001_lightNotesData.bin</dataPath>
        </tableStructure>
      </table>
    </container>
    <container>
      <label>Descriptive Statistics</label>
      <table commandName="Descriptives" subType="Descriptive Statistics"
             type="table">
        <tableStructure>
          <dataPath>00000000002_lightTableData.bin</dataPath>
        </tableStructure>
      </table>
    </container>
  </heading>
</heading>

• The heading Element
• The label Element
• The container Element
• The text Element (Inside container)
• The html Element
• The table Element
• The graph Element
• The model Element
• The object and image Elements
• The tree Element
• Path Elements
• The pageSetup Element
• The text Element (Inside pageParagraph)

heading[root_heading]
   :creator-version?
   :creator?
   :creation-date-time?
   :lockReader=bool?
   :schemaLocation?
=> label pageSetup? (container | heading)*

heading
   :creator-version?
   :commandName?
   :visibility[heading_visibility]=(collapsed)?
   :locale?
   :olang?
=> label (container | heading)*

label => TEXT

container
   :visibility=(visible | hidden)
   :page-break-before=(always)?
   :text-align=(left | center)?
   :width=dimension
=> label (table | container_text | graph | model | object | image | tree)

text[container_text]
  :type[text_type]=(title | log | text | page-title)
  :commandName?
  :creator-version?
=> html

html :lang=(en) => TEXT

table
   :VDPId?
   :ViZmlSource?
   :activePageId=int?
   :commandName
   :creator-version?
   :displayFiltering=bool?
   :maxNumCells=int?
   :orphanTolerance=int?
   :rowBreakNumber=int?
   :subType
   :tableId
   :tableLookId?
   :type[table_type]=(table | note | warning)
=> tableProperties? tableStructure

tableStructure => path? dataPath csvPath?

graph
   :VDPId?
   :ViZmlSource?
   :commandName?
   :creator-version?
   :dataMapId?
   :dataMapURI?
   :editor?
   :refMapId?
   :refMapURI?
   :csvFileIds?
   :csvFileNames?
=> dataPath? path csvPath?

model
   :PMMLContainerId?
   :PMMLId
   :StatXMLContainerId
   :VDPId
   :auxiliaryViewName
   :commandName
   :creator-version
   :mainViewName
=> ViZml? dataPath? path | pmmlContainerPath statsContainerPath

pmmlContainerPath => TEXT

statsContainerPath => TEXT

ViZml :viewName? => TEXT

object
   :commandName?
   :type[object_type]=(unknown)?
   :uri
=> EMPTY

image
   :commandName?
   :VDPId
=> dataPath

tree
   :commandName
   :creator-version
   :name
   :type
=> dataPath path

dataPath => TEXT

path => TEXT

csvPath => TEXT

pageSetup
   :initial-page-number=int?
   :chart-size=(as-is | full-height | half-height | quarter-height | OTHER)?
   :margin-left=dimension?
   :margin-right=dimension?
   :margin-top=dimension?
   :margin-bottom=dimension?
   :paper-height=dimension?
   :paper-width=dimension?
   :reference-orientation?
   :space-after=dimension?
=> pageHeader pageFooter

pageHeader => pageParagraph?

pageFooter => pageParagraph?

pageParagraph => pageParagraph_text

text[pageParagraph_text] :type=(title | text) => TEXT

<html xmlns="http://xml.spss.com/spss/viewer/viewer-tree">
    <head></head>
    <body>
        <p style="text-align:right; margin-top: 0">Page &[Page]</p>
    </body>
</html>

2.2 Light Detail Member Format ¶

This section describes the format of “light” detail .bin members. These members have a binary format which we describe here in terms of a context-free grammar using the following conventions:

NonTerminal ⇒ …

Nonterminals have CamelCaps names, and ⇒ indicates a production. The right-hand side of a production is often broken across multiple lines. Break points are chosen for aesthetics only and have no semantic significance.

00, 01, …, ff.

A bytes with a fixed value, written as a pair of hexadecimal digits.

i0, i1, …, i9, i10, i11, …

ib0, ib1, …, ib9, ib10, ib11, …

A 32-bit integer in little-endian or big-endian byte order, respectively, with a fixed value, written in decimal. Prefixed by ‘i’ for little-endian or ‘ib’ for big-endian.

byte

A byte.

bool

A byte with value 0 or 1.

int16

be16

A 16-bit unsigned integer in little-endian or big-endian byte order, respectively.

int32

be32

A 32-bit unsigned integer in little-endian or big-endian byte order, respectively.

int64

be64

A 64-bit unsigned integer in little-endian or big-endian byte order, respectively.

double

A 64-bit IEEE floating-point number.

float

A 32-bit IEEE floating-point number.

string

bestring

A 32-bit unsigned integer, in little-endian or big-endian byte order, respectively, followed by the specified number of bytes of character data. (The encoding is indicated by the Formats nonterminal.)

x?

x is optional, e.g. 00? is an optional zero byte.

x*n

x is repeated n times, e.g. byte*10 for ten arbitrary bytes.

x[name]

Gives x the specified name. Names are used in textual explanations. They are also used, also bracketed, to indicate counts, e.g. int32[n] byte*[n] for a 32-bit integer followed by the specified number of arbitrary bytes.

a | b

Either a or b.

(x)

Parentheses are used for grouping to make precedence clear, especially in the presence of |, e.g. in 00 (01 | 02 | 03) 00.

count(x)

becount(x)

A 32-bit unsigned integer, in little-endian or big-endian byte order, respectively, that indicates the number of bytes in x, followed by x itself.

v1(x)

In a version 1 .bin member, x; in version 3, nothing. (The .bin header indicates the version.)

v3(x)

In a version 3 .bin member, x; in version 1, nothing.

PSPP uses this grammar to parse light detail members. See src/output/spv/light-binary.grammar in the PSPP source tree for the full grammar.

Little-endian byte order is far more common in this format, but a few pieces of the format use big-endian byte order.

Light detail members express linear units in two ways: points (pt), at 72/inch, and “device-independent pixels” (px), at 96/inch. To convert from pt to px, multiply by 1.33 and round up. To convert from px to pt, divide by 1.33 and round down.

A “light” detail member .bin consists of a number of sections concatenated together, terminated by an optional byte 01:

Table =>
    Header Titles Footnotes
    Areas Borders PrintSettings TableSettings Formats
    Dimensions Axes Cells
    01?

The following sections go into more detail.

• Header
• Titles
• Footnotes
• Areas
• Borders
• Print Settings
• Table Settings
• Formats
• Dimensions
• Categories
• Axes
• Cells
• Value
• ValueMod

Header =>
    01 00
    (i1 | i3)[version]
    bool[x0]
    bool[x1]
    bool[rotate-inner-column-labels]
    bool[rotate-outer-row-labels]
    bool[x2]
    int32[x3]
    int32[min-col-heading-width] int32[max-col-heading-width]
    int32[min-row-heading-width] int32[max-row-heading-width]
    int64[table-id]

+------------------+-------------------------------------------------+
|                  |                  column headings                |
|                  +-------------------------------------------------+
|      corner      |                                                 |
|       and        |                                                 |
|   row headings   |                      data                       |
|                  |                                                 |
|                  |                                                 |
+------------------+-------------------------------------------------+

Titles =>
    Value[title] 01?
    Value[subtype] 01? 31
    Value[user-title] 01?
    (31 Value[corner-text] | 58)
    (31 Value[caption] | 58)

Footnotes => int32[n-footnotes] Footnote*[n-footnotes]
Footnote => Value[text] (58 | 31 Value[marker]) int32[show]

Areas => 00? Area*8
Area =>
    byte[index] 31
    string[typeface] float[size] int32[style] bool[underline]
    int32[halign] int32[valign]
    string[fg-color] string[bg-color]
    bool[alternate] string[alt-fg-color] string[alt-bg-color]
    v3(int32[left-margin] int32[right-margin] int32[top-margin] int32[bottom-margin])

Borders =>
    count(
        ib1[endian]
        be32[n-borders] Border*[n-borders]
        bool[show-grid-lines]
        00 00 00)

Border =>
    be32[border-type]
    be32[stroke-type]
    be32[color]

PrintSettings =>
    count(
        ib1[endian]
        bool[all-layers]
        bool[paginate-layers]
        bool[fit-width]
        bool[fit-length]
        bool[top-continuation]
        bool[bottom-continuation]
        be32[n-orphan-lines]
        bestring[continuation-string])

TableSettings =>
    count(
      v3(
        ib1[endian]
        be32[x5]
        be32[current-layer]
        bool[omit-empty]
        bool[show-row-labels-in-corner]
        bool[show-alphabetic-markers]
        bool[footnote-marker-superscripts]
        byte[x6]
        becount(
          Breakpoints[row-breaks] Breakpoints[column-breaks]
          Keeps[row-keeps] Keeps[column-keeps]
          PointKeeps[row-point-keeps] PointKeeps[column-point-keeps]
        )
        bestring[notes]
        bestring[table-look]
        )...)

Breakpoints => be32[n-breaks] be32*[n-breaks]

Keeps => be32[n-keeps] Keep*[n-keeps]
Keep => be32[offset] be32[n]

PointKeeps => be32[n-point-keeps] PointKeep*[n-point-keeps]
PointKeep => be32[offset] be32 be32

let current-layer = 0
for each i from d downto 1:
    current-layer = (n_i \times current-layer) + x_i

Formats =>
    int32[n-widths] int32*[n-widths]
    string[locale]
    int32[current-layer]
    bool[x7] bool[x8] bool[x9]
    Y0
    CustomCurrency
    count(
      v1(X0?)
      v3(count(X1 count(X2)) count(X3)))
Y0 => int32[epoch] byte[decimal] byte[grouping]
CustomCurrency => int32[n-ccs] string*[n-ccs]

X0 => byte*14 Y1 Y2
Y1 =>
    string[command] string[command-local]
    string[language] string[charset] string[locale]
    bool[x10] bool[include-leading-zero] bool[x12] bool[x13]
    Y0
Y2 => CustomCurrency byte[missing] bool[x17]

X1 =>
    bool[x14]
    byte[show-title]
    bool[x16]
    byte[lang]
    byte[show-variables]
    byte[show-values]
    int32[x18] int32[x19]
    00*17
    bool[x20]
    bool[show-caption]

X2 =>
    int32[n-row-heights] int32*[n-row-heights]
    int32[n-style-map] StyleMap*[n-style-map]
    int32[n-styles] StylePair*[n-styles]
    count((i0 i0)?)
StyleMap => int64[cell-index] int16[style-index]

X3 =>
    01 00 byte[x21] 00 00 00
    Y1
    double[small] 01
    (string[dataset] string[datafile] i0 int32[date] i0)?
    Y2
    (int32[x22] i0 01?)?

Dimensions => int32[n-dims] Dimension*[n-dims]
Dimension =>
    Value[name] DimProperties
    int32[n-categories] Category*[n-categories]
DimProperties =>
    byte[x1]
    byte[x2]
    int32[x3]
    bool[hide-dim-label]
    bool[hide-all-labels]
    01 int32[dim-index]

Category => Value[name] (Leaf | Group)
Leaf => 00 00 00 i2 int32[leaf-index] i0
Group =>
    bool[merge] 00 01 int32[x23]
    i-1 int32[n-subcategories] Category*[n-subcategories]

Axes =>
    int32[n-layers] int32[n-rows] int32[n-columns]
    int32*[n-layers] int32*[n-rows] int32*[n-columns]

Cells => int32[n-cells] Cell*[n-cells]
Cell => int64[index] v1(00?) Value

let index = 0
for each i from 1 to d:
    index = (n_i \times index) + x_i

Value => 00? 00? 00? 00? RawValue
RawValue =>
    01 ValueMod int32[format] double[x]
  | 02 ValueMod int32[format] double[x]
    string[var-name] string[value-label] byte[show]
  | 03 string[local] ValueMod string[id] string[c] bool[fixed]
  | 04 ValueMod int32[format] string[value-label] string[var-name]
    byte[show] string[s]
  | 05 ValueMod string[var-name] string[var-label] byte[show]
  | 06 string[local] ValueMod string[id] string[c]
  | ValueMod string[template] int32[n-args] Argument*[n-args]
Argument =>
    i0 Value
  | int32[x] i0 Value*[x]      /* x > 0 */

ValueMod =>
    58
  | 31
    int32[n-refs] int16*[n-refs]
    int32[n-subscripts] string*[n-subscripts]
    v1(00 (i1 | i2) 00? 00? int32 00? 00?)
    v3(count(TemplateString StylePair))

TemplateString => count((count((i0 (58 | 31 55))?) (58 | 31 string[id]))?)

StylePair =>
    (31 FontStyle | 58)
    (31 CellStyle | 58)

FontStyle =>
    bool[bold] bool[italic] bool[underline] bool[show]
    string[fg-color] string[bg-color]
    string[typeface] byte[size]

CellStyle =>
    int32[halign] int32[valign] double[decimal-offset]
    int16[left-margin] int16[right-margin]
    int16[top-margin] int16[bottom-margin]

2.3 Legacy Detail Member Binary Format ¶

Whereas the light binary format represents everything about a given pivot table, the legacy binary format conceptually consists of a number of named sources, each of which consists of a number of named variables, each of which is a 1-dimensional array of numbers or strings or a mix. Thus, the legacy binary member format is quite simple.

This section uses the same context-free grammar notation as in the previous section, with the following additions:

vAF(x)

In a version 0xaf legacy member, x; in other versions, nothing. (The legacy member header indicates the version; see below.)

vB0(x)

In a version 0xb0 legacy member, x; in other versions, nothing.

A legacy detail member .bin has the following overall format:

LegacyBinary =>
    00 byte[version] int16[n-sources] int32[member-size]
    Metadata*[n-sources]
    #Data*[n-sources]
    #Strings?

version is a version number that affects the interpretation of some of the other data in the member. Versions 0xaf and 0xb0 are known. We will refer to “version 0xaf” and “version 0xb0” members later on.

A legacy member consists of n-sources data sources, each of which has Metadata and Data.

member-size is the size of the legacy binary member, in bytes.

The Data and Strings above are commented out because the Metadata has some oddities that mean that the Data sometimes seems to start at an unexpected place. The following section goes into detail.

• Metadata
• Numeric Data
• String Data

Metadata =>
    int32[n-values] int32[n-variables] int32[data-offset]
    vAF(byte*28[source-name])
    vB0(byte*64[source-name] int32[x])

Data => Variable*[n-variables]
Variable => byte*288[variable-name] double*[n-values]

Strings => SourceMaps[maps] Labels

SourceMaps => int32[n-maps] SourceMap*[n-maps]

SourceMap => string[source-name] int32[n-variables] VariableMap*[n-variables]
VariableMap => string[variable-name] int32[n-data] DatumMap*[n-data]
DatumMap => int32[value-idx] int32[label-idx]

Labels => int32[n-labels] Label*[n-labels]
Label => int32[frequency] string[label]

2.4 Legacy Detail Member XML Format ¶

The design of the detail XML format is not what one would end up with for describing pivot tables. This is because it is a special case of a much more general format (“visualization XML” or “VizML”) that can describe a wide range of visualizations. Most of this generality is overkill for tables, and so we end up with a funny subset of a general-purpose format.

An XML Schema for VizML is available, distributed with SPSS binaries, under a nonfree license. It contains documentation that is occasionally helpful.

This section describes the detail XML format using the same notation already used for the structure XML format (see Structure Member Format). See src/output/spv/detail-xml.grammar in the PSPP source tree for the full grammar that it uses for parsing.

The important elements of the detail XML format are:

• Variables. See Variable Elements.
• Assignment of variables to axes. A variable can appear as columns, or rows, or layers. The faceting element and its sub-elements describe this assignment.
• Styles and other annotations.

This description is not detailed enough to write legacy tables. Instead, write tables in the light binary format.

• The visualization Element
• Variable Elements
• The extension Element
• The graph Element
• The location Element
• The faceting Element
• The facetLayout Element
• The label Element
• The setCellProperties Element
• The setFormat Element
• The interval Element
• The style Element
• The labelFrame Element
• Legacy Properties

visualization
   :creator
   :date
   :lang
   :name
   :style[style_ref]=ref style
   :type
   :version
   :schemaLocation?
=> visualization_extension?
   userSource
   (sourceVariable | derivedVariable)+
   categoricalDomain?
   graph
   labelFrame[lf1]*
   container?
   labelFrame[lf2]*
   style+
   layerController?

extension[visualization_extension]
   :numRows=int?
   :showGridline=bool?
   :minWidthSet=(true)?
   :maxWidthSet=(true)?
=> EMPTY

userSource :missing=(listwise | pairwise)? => EMPTY

categoricalDomain => variableReference simpleSort

simpleSort :method[sort_method]=(custom) => categoryOrder

container :style=ref style => container_extension? location+ labelFrame*

extension[container_extension] :combinedFootnotes=(true) => EMPTY

layerController
   :source=(tableData)
   :target=ref label?
=> EMPTY

<derivedVariable categorical="true" dependsOn="dimension0categories"
                 id="dimension0group0map" value="map(dimension0group0)">
  <stringFormat>
    <relabel from="2" to="Input"/>
    <relabel from="10" to="Missing Value Handling"/>
    <relabel from="14" to="Resources"/>
    <relabel from="0" to=""/>
    <relabel from="1" to=""/>
    <relabel from="13" to=""/>
  </stringFormat>
  <valueMapEntry from="2;3;5;6;7;8;9" to="2"/>
  <valueMapEntry from="10;11" to="10"/>
  <valueMapEntry from="14;15" to="14"/>
  <valueMapEntry from="0" to="0"/>
  <valueMapEntry from="1" to="1"/>
  <valueMapEntry from="13" to="13"/>
</derivedVariable>

sourceVariable
   :id
   :categorical=(true)
   :source
   :domain=ref categoricalDomain?
   :sourceName
   :dependsOn=ref sourceVariable?
   :label?
   :labelVariable=ref sourceVariable?
=> variable_extension* (format | stringFormat)?

derivedVariable
   :id
   :categorical=(true)
   :value
   :dependsOn=ref sourceVariable?
=> variable_extension* (format | stringFormat)? valueMapEntry*

valueMapEntry :from :to => EMPTY

extension[container_extension] :combinedFootnotes=(true) => EMPTY

extension[variable_extension] :from :helpId => EMPTY

<extension from="0" helpId="corrected_model"/>
<extension from="3" helpId="error"/>
<extension from="4" helpId="total_9"/>
<extension from="5" helpId="corrected_total"/>

<extension from="0" helpId="notes"/>
<extension from="1" helpId="notes"/>
<extension from="2" helpId="notes"/>
<extension from="5" helpId="notes"/>
<extension from="6" helpId="notes"/>
<extension from="7" helpId="notes"/>
<extension from="8" helpId="notes"/>
<extension from="12" helpId="notes"/>
<extension from="13" helpId="no_help"/>
<extension from="14" helpId="notes"/>

graph
   :cellStyle=ref style
   :style=ref style
=> location+ coordinates faceting facetLayout interval

coordinates => EMPTY

location
   :part=(height | width | top | bottom | left | right)
   :method=(sizeToContent | attach | fixed | same)
   :min=dimension?
   :max=dimension?
   :target=ref (labelFrame | graph | container)?
   :value?
=> EMPTY

faceting => layer[layers1]* cross layer[layers2]*

cross => (unity | nest) (unity | nest)

unity => EMPTY

nest => variableReference[vars]+

variableReference :ref=ref (sourceVariable | derivedVariable) => EMPTY

layer
   :variable=ref (sourceVariable | derivedVariable)
   :value
   :visible=bool?
   :method[layer_method]=(nest)?
   :titleVisible=bool?
=> EMPTY

<nest>
  <variableReference ref="dimension0categories"/>
  <variableReference ref="dimension0group0"/>
  <variableReference ref="dimension0"/>
</nest>

<nest>
  <variableReference ref="dimension1categories"/>
  <variableReference ref="dimension1group0"/>
  <variableReference ref="dimension1"/>
  <variableReference ref="dimension0categories"/>
  <variableReference ref="dimension0"/>
</nest>

<layer value="0" variable="dimension0categories" visible="true"/>
<layer value="dimension0" variable="dimension0" visible="false"/>

facetLayout => tableLayout setCellProperties[scp1]*
               facetLevel+ setCellProperties[scp2]*

tableLayout
   :verticalTitlesInCorner=bool
   :style=ref style?
   :fitCells=(ticks both)?
=> EMPTY

facetLevel :level=int :gap=dimension? => axis

axis :style=ref style => label? majorTicks

majorTicks
   :labelAngle=int
   :length=dimension
   :style=ref style
   :tickFrameStyle=ref style
   :labelFrequency=int?
   :stagger=bool?
=> gridline?

gridline
   :style=ref style
   :zOrder=int
=> EMPTY

label
   :style=ref style
   :textFrameStyle=ref style?
   :purpose=(title | subTitle | subSubTitle | layer | footnote)?
=> text+ | descriptionGroup

descriptionGroup
   :target=ref faceting
   :separator?
=> (description | text)+

description :name=(variable | value) => EMPTY

text
   :usesReference=int?
   :definesReference=int?
   :position=(subscript | superscript)?
   :style=ref style
=> TEXT

<description name="value"/>

<description name="variable"/><text>:</text><description name="value"/>

setCellProperties
   :applyToConverse=bool?
=> (setStyle | setFrameStyle | setFormat | setMetaData)* union[union_]?

union => intersect+

intersect => where+ | intersectWhere | alternating | EMPTY

where
   :variable=ref (sourceVariable | derivedVariable)
   :include
=> EMPTY

intersectWhere
   :variable=ref (sourceVariable | derivedVariable)
   :variable2=ref (sourceVariable | derivedVariable)
=> EMPTY

alternating => EMPTY

setStyle
   :target=ref (labeling | graph | interval | majorTicks)
   :style=ref style
=> EMPTY

setMetaData :target=ref graph :key :value => EMPTY

setFormat
   :target=ref (majorTicks | labeling)
   :reset=bool?
=> format | numberFormat | stringFormat+ | dateTimeFormat | elapsedTimeFormat

setFrameStyle
   :style=ref style
   :target=ref majorTicks
=> EMPTY

setStyle
   :target=ref (labeling | graph | interval | majorTicks)
   :style=ref style
=> EMPTY

setFormat
   :target=ref (majorTicks | labeling)
   :reset=bool?
=> format | numberFormat | stringFormat+ | dateTimeFormat | elapsedTimeFormat

numberFormat
   :minimumIntegerDigits=int?
   :maximumFractionDigits=int?
   :minimumFractionDigits=int?
   :useGrouping=bool?
   :scientific=(onlyForSmall | whenNeeded | true | false)?
   :small=real?
   :prefix?
   :suffix?
=> affix*

stringFormat => relabel* affix*

relabel :from=real :to => EMPTY

dateTimeFormat
   :baseFormat[dt_base_format]=(date | time | dateTime)
   :separatorChars?
   :mdyOrder=(dayMonthYear | monthDayYear | yearMonthDay)?
   :showYear=bool?
   :yearAbbreviation=bool?
   :showQuarter=bool?
   :quarterPrefix?
   :quarterSuffix?
   :showMonth=bool?
   :monthFormat=(long | short | number | paddedNumber)?
   :showWeek=bool?
   :weekPadding=bool?
   :weekSuffix?
   :showDayOfWeek=bool?
   :dayOfWeekAbbreviation=bool?
   :dayPadding=bool?
   :dayOfMonthPadding=bool?
   :hourPadding=bool?
   :minutePadding=bool?
   :secondPadding=bool?
   :showDay=bool?
   :showHour=bool?
   :showMinute=bool?
   :showSecond=bool?
   :showMillis=bool?
   :dayType=(month | year)?
   :hourFormat=(AMPM | AS_24 | AS_12)?
=> affix*

elapsedTimeFormat
   :baseFormat[dt_base_format]=(date | time | dateTime)
   :dayPadding=bool?
   :hourPadding=bool?
   :minutePadding=bool?
   :secondPadding=bool?
   :showYear=bool?
   :showDay=bool?
   :showHour=bool?
   :showMinute=bool?
   :showSecond=bool?
   :showMillis=bool?
=> affix*

format
   :baseFormat[f_base_format]=(date | time | dateTime | elapsedTime)?
   :errorCharacter?
   :separatorChars?
   :mdyOrder=(dayMonthYear | monthDayYear | yearMonthDay)?
   :showYear=bool?
   :showQuarter=bool?
   :quarterPrefix?
   :quarterSuffix?
   :yearAbbreviation=bool?
   :showMonth=bool?
   :monthFormat=(long | short | number | paddedNumber)?
   :dayPadding=bool?
   :dayOfMonthPadding=bool?
   :showWeek=bool?
   :weekPadding=bool?
   :weekSuffix?
   :showDayOfWeek=bool?
   :dayOfWeekAbbreviation=bool?
   :hourPadding=bool?
   :minutePadding=bool?
   :secondPadding=bool?
   :showDay=bool?
   :showHour=bool?
   :showMinute=bool?
   :showSecond=bool?
   :showMillis=bool?
   :dayType=(month | year)?
   :hourFormat=(AMPM | AS_24 | AS_12)?
   :minimumIntegerDigits=int?
   :maximumFractionDigits=int?
   :minimumFractionDigits=int?
   :useGrouping=bool?
   :scientific=(onlyForSmall | whenNeeded | true | false)?
   :small=real?
   :prefix?
   :suffix?
   :tryStringsAsNumbers=bool?
   :negativesOutside=bool?
=> relabel* affix*

affix
   :definesReference=int
   :position=(subscript | superscript)
   :suffix=bool
   :value
=> EMPTY

interval :style=ref style => labeling footnotes?

labeling
   :style=ref style?
   :variable=ref (sourceVariable | derivedVariable)
=> (formatting | format | footnotes)*

formatting :variable=ref (sourceVariable | derivedVariable) => formatMapping*

formatMapping :from=int => format?

footnotes
   :superscript=bool?
   :variable=ref (sourceVariable | derivedVariable)
=> footnoteMapping*

footnoteMapping :definesReference=int :from=int :to => EMPTY

style
   :color=color?
   :color2=color?
   :labelAngle=real?
   :border-bottom=(solid | thick | thin | double | none)?
   :border-top=(solid | thick | thin | double | none)?
   :border-left=(solid | thick | thin | double | none)?
   :border-right=(solid | thick | thin | double | none)?
   :border-bottom-color?
   :border-top-color?
   :border-left-color?
   :border-right-color?
   :font-family?
   :font-size?
   :font-weight=(regular | bold)?
   :font-style=(regular | italic)?
   :font-underline=(none | underline)?
   :margin-bottom=dimension?
   :margin-left=dimension?
   :margin-right=dimension?
   :margin-top=dimension?
   :textAlignment=(left | right | center | decimal | mixed)?
   :labelLocationHorizontal=(positive | negative | center)?
   :labelLocationVertical=(positive | negative | center)?
   :decimal-offset=dimension?
   :size?
   :width?
   :visible=bool?
=> EMPTY

labelFrame :style=ref style => location+ label? paragraph?

paragraph :hangingIndent=dimension? => EMPTY

tableProperties
   :name?
=> generalProperties footnoteProperties cellFormatProperties borderProperties printingProperties

generalProperties
   :hideEmptyRows=bool?
   :maximumColumnWidth=dimension?
   :maximumRowWidth=dimension?
   :minimumColumnWidth=dimension?
   :minimumRowWidth=dimension?
   :rowDimensionLabels=(inCorner | nested)?
=> EMPTY

footnoteProperties
   :markerPosition=(superscript | subscript)?
   :numberFormat=(alphabetic | numeric)?
=> EMPTY

cellFormatProperties => cell_style+

any[cell_style]
   :alternatingColor=color?
   :alternatingTextColor=color?
=> style

style
   :color=color?
   :color2=color?
   :font-family?
   :font-size?
   :font-style=(regular | italic)?
   :font-weight=(regular | bold)?
   :font-underline=(none | underline)?
   :labelLocationVertical=(positive | negative | center)?
   :margin-bottom=dimension?
   :margin-left=dimension?
   :margin-right=dimension?
   :margin-top=dimension?
   :textAlignment=(left | right | center | decimal | mixed)?
   :decimal-offset=dimension?
=> EMPTY

borderProperties => border_style+

any[border_style]
   :borderStyleType=(none | solid | dashed | thick | thin | double)?
   :color=color?
=> EMPTY

printingProperties
   :printAllLayers=bool?
   :rescaleLongTableToFitPage=bool?
   :rescaleWideTableToFitPage=bool?
   :windowOrphanLines=int?
   :continuationText?
   :continuationTextAtBottom=bool?
   :continuationTextAtTop=bool?
   :printEachLayerOnSeparatePage=bool?
=> EMPTY

3.1 The .stt Format ¶

The .stt file format is an XML file that contains a subset of the SPV structure member format (see Structure Member Format). Its root element is a tableProperties element (see Legacy Properties).

3.2 The .tlo Format ¶

A .tlo file has a custom binary format. This section describes it using the syntax used previously for SPV binary members (see Light Detail Member Format). There is one new convention: TLO files express colors as int32 values in which the low 8 bits are the red component, the next 8 bits are green, and next 8 bits are blue, and the high bits are zeros.

TLO files support various features that SPV files do not. PSPP implements the SPV feature set, so it mostly ignores the added TLO features. The details of this mapping are explained below.

At the top level, a TLO file consists of five sections. The first four are always present and the last one is optional:

TableLook =>
   PTTableLook[tl]
   PVSeparatorStyle[ss]
   PVCellStyle[cs]
   PVTextStyle[ts]
   V2Styles?

Each section is described below.

• PTTableLook
• PVSeparatorStyle
• PVCellStyle and PVTextStyle
• V2Styles

PTTableLook =>
   ff ff 00 00 "PTTableLook" (00|02)[version]
   int16[flags]
   00 00
   bool[nested-row-labels] 00
   bool[footnote-marker-subscripts] 00
   i54 i18

PVSeparatorStyle =>
   ff ff 00 00 "PVSeparatorStyle" 00
   Separator*4[sep1]
   03 80 00
   Separator*4[sep2]

Separator =>
   case(
       00 00
     | 01 00 int32[color] int16[style] int16[width]
   )[type]

PVCellStyle =>
   ff ff 00 00 "PVCellStyle"
   AreaColor[title-color]

PVTextStyle =>
   ff ff 00 00 "PVTextStyle" 00
   AreaStyle[title-style] MostAreas*7[most-areas]

MostAreas =>
   06 80
   AreaColor[color] 08 80 00 AreaStyle[style]

AreaColor =>
   00 01 00 int32[color10] int32[color0] byte[shading] 00

AreaStyle =>
   int16[valign] int16[halign] int16[decimal-offset]
   int16[left-margin] int16[right-margin] int16[top-margin] int16[bottom-margin]
   00 00 01 00
   int32[font-size] int16[stretch]
   00*2
   int32[rotation-angle]
   00*4
   int16[weight]
   00*2
   bool[italic] bool[underline] bool[strikethrough]
   int32[rtf-charset-number]
   byte[x]
   byte[font-name-len] byte*[font-name-len][font-name]
   int32[text-color]
   00*2

V2Styles =>
   Separator*11[sep3]
   byte[continuation-len] byte*[continuation-len][continuation]
   int32[min-col-width] int32[max-col-width]
   int32[min-row-height] int32[max-row-height]

4.1 Common Wrapper Format ¶

An encrypted file wrapper begins with the following 36-byte header, where xxx identifies the type of file encapsulated: SAV for a system file, SPS for a syntax file, SPV for a viewer file. PSPP code for identifying these files just checks for the ENCRYPTED keyword at offset 8, but the other bytes are also fixed in practice:

0000  1c 00 00 00 00 00 00 00  45 4e 43 52 59 50 54 45  |........ENCRYPTE|
0010  44 xx xx xx 15 00 00 00  00 00 00 00 00 00 00 00  |Dxxx............|
0020  00 00 00 00                                       |....|

Following the fixed header is essentially the regular contents of the encapsulated file in its usual format, with each 16-byte block encrypted with AES-256 in ECB mode.

To make the plaintext an even multiple of 16 bytes in length, the encryption process appends PKCS #7 padding, as specified in RFC 5652 section 6.3. Padding appends 1 to 16 bytes to the plaintext, in which each byte of padding is the number of padding bytes added. If the plaintext is, for example, 2 bytes short of a multiple of 16, the padding is 2 bytes with value 02; if the plaintext is a multiple of 16 bytes in length, the padding is 16 bytes with value 0x10.

The AES-256 key is derived from a password in the following way:

1. Start from the literal password typed by the user. Truncate it to at most 10 bytes, then append as many null bytes as necessary until there are exactly 32 bytes. Call this password.
2. Let constant be the following 73-byte constant:

   0000  00 00 00 01 35 27 13 cc  53 a7 78 89 87 53 22 11
   0010  d6 5b 31 58 dc fe 2e 7e  94 da 2f 00 cc 15 71 80
   0020  0a 6c 63 53 00 38 c3 38  ac 22 f3 63 62 0e ce 85
   0030  3f b8 07 4c 4e 2b 77 c7  21 f5 1a 80 1d 67 fb e1
   0040  e1 83 07 d8 0d 00 00 01  00
   

3. Compute CMAC-AES-256(password, constant). Call the 16-byte result cmac.
4. The 32-byte AES-256 key is cmac || cmac, that is, cmac repeated twice.

Example ¶

Consider the password ‘pspp’. password is:

0000  70 73 70 70 00 00 00 00  00 00 00 00 00 00 00 00  |pspp............|
0010  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|

cmac is:

0000  3e da 09 8e 66 04 d4 fd  f9 63 0c 2c a8 6f b0 45

The AES-256 key is:

0000  3e da 09 8e 66 04 d4 fd  f9 63 0c 2c a8 6f b0 45
0010  3e da 09 8e 66 04 d4 fd  f9 63 0c 2c a8 6f b0 45

• Checking Passwords

4.2 Password Encoding ¶

SPSS also supports what it calls “encrypted passwords.” These are not encrypted. They are encoded with a simple, fixed scheme. An encoded password is always a multiple of 2 characters long, and never longer than 20 characters. The characters in an encoded password are always in the graphic ASCII range 33 through 126. Each successive pair of characters in the password encodes a single byte in the plaintext password.

Use the following algorithm to decode a pair of characters:

1. Let a be the ASCII code of the first character, and b be the ASCII code of the second character.
2. Let ah be the most significant 4 bits of a. Find the line in the table below that has ah on the left side. The right side of the line is a set of possible values for the most significant 4 bits of the decoded byte.

   2  ⇒ 2367
   3  ⇒ 0145
   47 ⇒ 89cd
   56 ⇒ abef
   

3. Let bh be the most significant 4 bits of b. Find the line in the second table below that has bh on the left side. The right side of the line is a set of possible values for the most significant 4 bits of the decoded byte. Together with the results of the previous step, only a single possibility is left.

   2  ⇒ 139b
   3  ⇒ 028a
   47 ⇒ 46ce
   56 ⇒ 57df
   

4. Let al be the least significant 4 bits of a. Find the line in the table below that has al on the left side. The right side of the line is a set of possible values for the least significant 4 bits of the decoded byte.

   03cf ⇒ 0145
   12de ⇒ 2367
   478b ⇒ 89cd
   569a ⇒ abef
   

5. Let bl be the least significant 4 bits of b. Find the line in the table below that has bl on the left side. The right side of the line is a set of possible values for the least significant 4 bits of the decoded byte. Together with the results of the previous step, only a single possibility is left.

   03cf ⇒ 028a
   12de ⇒ 139b
   478b ⇒ 46ce
   569a ⇒ 57df
   

Example ¶

Consider the encoded character pair ‘-|’. a is 0x2d and b is 0x7c, so ah is 2, bh is 7, al is 0xd, and bl is 0xc. ah means that the most significant four bits of the decoded character is 2, 3, 6, or 7, and bh means that they are 4, 6, 0xc, or 0xe. The single possibility in common is 6, so the most significant four bits are 6. Similarly, al means that the least significant four bits are 2, 3, 6, or 7, and bl means they are 0, 2, 8, or 0xa, so the least significant four bits are 2. The decoded character is therefore 0x62, the letter ‘b’.

5.1 Portable File Characters ¶

Portable files are arranged as a series of lines of 80 characters each. Each line is terminated by a carriage-return, line-feed sequence (“new-lines”). New-lines are only used to avoid line length limits imposed by some OSes; they are not meaningful.

Most lines in portable files are exactly 80 characters long. The only exception is a line that ends in one or more spaces, in which the spaces may optionally be omitted. Thus, a portable file reader must act as though a line shorter than 80 characters is padded to that length with spaces.

The file must be terminated with a ‘Z’ character. In addition, if the final line in the file does not have exactly 80 characters, then it is padded on the right with ‘Z’ characters. (The file contents may be in any character set; the file contains a description of its own character set, as explained in the next section. Therefore, the ‘Z’ character is not necessarily an ASCII ‘Z’.)

For the rest of the description of the portable file format, new-lines and the trailing ‘Z’s will be ignored, as if they did not exist, because they are not an important part of understanding the file contents.

5.2 Portable File Structure ¶

Every portable file consists of the following records, in sequence:

• File header.
• Version and date info.
• Product identification.
• Author identification (optional).
• Subproduct identification (optional).
• Variable count.
• Case weight variable (optional).
• Variables. Each variable record may optionally be followed by a missing value record and a variable label record.
• Value labels (optional).
• Documents (optional).
• Data.

Most records are identified by a single-character tag code. The file header and version info record do not have a tag.

Other than these single-character codes, there are three types of fields in a portable file: floating-point, integer, and string. Floating-point fields have the following format:

• Zero or more leading spaces.
• Optional asterisk (‘*’), which indicates a missing value. The asterisk must be followed by a single character, generally a period (‘.’), but it appears that other characters may also be possible. This completes the specification of a missing value.
• Optional minus sign (‘-’) to indicate a negative number.
• A whole number, consisting of one or more base-30 digits: ‘0’ through ‘9’ plus capital letters ‘A’ through ‘T’.
• Optional fraction, consisting of a radix point (‘.’) followed by one or more base-30 digits.
• Optional exponent, consisting of a plus or minus sign (‘+’ or ‘-’) followed by one or more base-30 digits.
• A forward slash (‘/’).

Integer fields take a form identical to floating-point fields, but they may not contain a fraction.

String fields take the form of an integer field having value n, followed by exactly n characters, which are the string content.

5.3 Portable File Header ¶

Every portable file begins with a 464-byte header, consisting of a 200-byte collection of vanity splash strings, followed by a 256-byte character set translation table, followed by an 8-byte tag string.

The 200-byte segment is divided into five 40-byte sections, each of which represents the string charset SPSS PORT FILE in a different character set encoding, where charset is the name of the character set used in the file, e.g. ASCII or EBCDIC. Each string is padded on the right with spaces in its respective character set.

It appears that these strings exist only to inform those who might view the file on a screen, and that they are not parsed by SPSS products. Thus, they can be safely ignored. For those interested, the strings are supposed to be in the following character sets, in the specified order: EBCDIC, 7-bit ASCII, CDC 6-bit ASCII, 6-bit ASCII, Honeywell 6-bit ASCII.

The 256-byte segment describes a mapping from the character set used in the portable file to an arbitrary character set having characters at the following positions:

0–60

Control characters. Not important enough to describe in full here.

61–63

Reserved.

64–73

Digits ‘0’ through ‘9’.

74–99

Capital letters ‘A’ through ‘Z’.

100–125

Lowercase letters ‘a’ through ‘z’.

126

Space.

127–130

Symbols .<(+

131

Solid vertical pipe.

132–142

Symbols &[]!$*);^-/

143

Broken vertical pipe.

144–150

Symbols ,%_>?`:

151

British pound symbol.

152–155

Symbols @'=".

156

Less than or equal symbol.

157

Empty box.

158

Plus or minus.

159

Filled box.

160

Degree symbol.

161

Dagger.

162

Symbol ‘~’.

163

En dash.

164

Lower left corner box draw.

165

Upper left corner box draw.

166

Greater than or equal symbol.

167–176

Superscript ‘0’ through ‘9’.

177

Lower right corner box draw.

178

Upper right corner box draw.

179

Not equal symbol.

180

Em dash.

181

Superscript ‘(’.

182

Superscript ‘)’.

183

Horizontal dagger (?).

184–186

Symbols ‘{}\’.

187

Cents symbol.

188

Centered dot, or bullet.

189–255

Reserved.

Symbols that are not defined in a particular character set are set to the same value as symbol 64; i.e., to ‘0’.

The 8-byte tag string consists of the exact characters SPSSPORT in the portable file’s character set, which can be used to verify that the file is indeed a portable file.

5.4 Version and Date Info Record ¶

This record does not have a tag code. It has the following structure:

• A single character identifying the file format version. The letter A represents version 0, and so on.
• An 8-character string field giving the file creation date in the format YYYYMMDD.
• A 6-character string field giving the file creation time in the format HHMMSS.

5.5 Identification Records ¶

The product identification record has tag code ‘1’. It consists of a single string field giving the name of the product that wrote the portable file.

The author identification record has tag code ‘2’. It is optional. If present, it consists of a single string field giving the name of the person who caused the portable file to be written.

The subproduct identification record has tag code ‘3’. It is optional. If present, it consists of a single string field giving additional information on the product that wrote the portable file.

5.6 Variable Count Record ¶

The variable count record has tag code ‘4’. It consists of a single integer field giving the number of variables in the file dictionary.

5.7 Precision Record ¶

The precision record has tag code ‘5’. It consists of a single integer field specifying the maximum number of base-30 digits used in data in the file.

5.8 Case Weight Variable Record ¶

The case weight variable record is optional. If it is present, it indicates the variable used for weighting cases; if it is absent, cases are unweighted. It has tag code ‘6’. It consists of a single string field that names the weighting variable.

5.9 Variable Records ¶

Each variable record represents a single variable. Variable records have tag code ‘7’. They have the following structure:

• Width (integer). This is 0 for a numeric variable, and a number between 1 and 255 for a string variable.
• Name (string). 1–8 characters long. Must be in all capitals.

  A few portable files that contain duplicate variable names have been spotted in the wild. PSPP handles these by renaming the duplicates with numeric extensions: var_1, var_2, and so on.

• Print format. This is a set of three integer fields:
  • Format type (see Variable Record).
  • Format width. 1–40.
  • Number of decimal places. 1–40.

  A few portable files with invalid format types or formats that are not of the appropriate width for their variables have been spotted in the wild. PSPP assigns a default F or A format to a variable with an invalid format.

• Write format. Same structure as the print format described above.

Each variable record can optionally be followed by a missing value record, which has tag code ‘8’. A missing value record has one field, the missing value itself (a floating-point or string, as appropriate). Up to three of these missing value records can be used.

There is also a record for missing value ranges, which has tag code ‘B’. It is followed by two fields representing the range, which are floating-point or string as appropriate. If a missing value range is present, it may be followed by a single missing value record.

Tag codes ‘9’ and ‘A’ represent LO THRU x and x THRU HI ranges, respectively. Each is followed by a single field representing x. If one of the ranges is present, it may be followed by a single missing value record.

In addition, each variable record can optionally be followed by a variable label record, which has tag code ‘C’. A variable label record has one field, the variable label itself (string).

5.10 Value Label Records ¶

Value label records have tag code ‘D’. They have the following format:

• Variable count (integer).
• List of variables (strings). The variable count specifies the number in the list. Variables are specified by their names. All variables must be of the same type (numeric or string), but string variables do not necessarily have the same width.
• Label count (integer).
• List of (value, label) tuples. The label count specifies the number of tuples. Each tuple consists of a value, which is numeric or string as appropriate to the variables, followed by a label (string).

A few portable files that specify duplicate value labels, that is, two different labels for a single value of a single variable, have been spotted in the wild. PSPP uses the last value label specified in these cases.

5.11 Document Record ¶

One document record may optionally follow the value label record. The document record consists of tag code ‘E’, following by the number of document lines as an integer, followed by that number of strings, each of which represents one document line. Document lines must be 80 bytes long or shorter.

5.12 Portable File Data ¶

The data record has tag code ‘F’. There is only one tag for all the data; thus, all the data must follow the dictionary. The data is terminated by the end-of-file marker ‘Z’, which is not valid as the beginning of a data element.

Data elements are output in the same order as the variable records describing them. String variables are output as string fields, and numeric variables are output as floating-point fields.

6.1 Record 0: Main Header Record ¶

All files in the corpus have this record at offset 0x100 with length 0xb0 (but readers should find this record, like the others, via the records table in the directory). Its format is:

uint16              one0;
char                product[62];
flt64               sysmis;
uint32              zero0;
uint32              zero1;
uint16              one1;
uint16              compressed;
uint16              nominal_case_size;
uint16              n_cases0;
uint16              weight_index;
uint16              zero2;
uint16              n_cases1;
uint16              zero3;
char                creation_date[8];
char                creation_time[8];
char                label[64];

uint16 one0;

uint16 one1;

Always set to 1.

uint32 zero0;

uint32 zero1;

uint16 zero2;

uint16 zero3;

Always set to 0.

It seems likely that one of these variables is set to 1 if weighting is enabled, but none of the files in the corpus is weighted.

char product[62];

Name of the program that created the file. Only the following unique values have been observed, in each case padded on the right with spaces:

DESPSS/PC+ System File Written by Data Entry II
PCSPSS SYSTEM FILE.  IBM PC DOS, SPSS/PC+
PCSPSS SYSTEM FILE.  IBM PC DOS, SPSS/PC+ V3.0
PCSPSS SYSTEM FILE.  IBM PC DOS, SPSS for Windows

Thus, it is reasonable to use the presence of the string ‘SPSS’ at offset 0x104 as a simple test for an SPSS/PC+ data file.

flt64 sysmis;

The system-missing value, as described previously (see SPSS/PC+ System File Format).

uint16 compressed;

Set to 0 if the data in the file is not compressed, 1 if the data is compressed with simple bytecode compression.

uint16 nominal_case_size;

Number of data elements per case. This is the number of variables, except that long string variables add extra data elements (one for every 8 bytes after the first 8). String variables in SPSS/PC+ system files are limited to 255 bytes.

uint16 n_cases0;

uint16 n_cases1;

The number of cases in the data record. Both values are the same. Some files in the corpus contain data for the number of cases noted here, followed by garbage that somewhat resembles data.

uint16 weight_index;

0, if the file is unweighted, otherwise a 1-based index into the data record of the weighting variable, e.g. 4 for the first variable after the 3 system-defined variables.

char creation_date[8];

The date that the file was created, in ‘mm/dd/yy’ format. Single-digit days and months are not prefixed by zeros. The string is padded with spaces on right or left or both, e.g. ‘_2/4/93_’, ‘10/5/87_’, and ‘_1/11/88’ (with ‘_’ standing in for a space) are all actual examples from the corpus.

char creation_time[8];

The time that the file was created, in ‘HH:MM:SS’ format. Single-digit hours are padded on a left with a space. Minutes and seconds are always written as two digits.

char file_label[64];

File label declared by the user, if any (see FILE LABEL in PSPP Users Guide). Padded on the right with spaces.

6.2 Record 1: Variables Record ¶

The variables record most commonly starts at offset 0x1b0, but it can be placed elsewhere. The record contains instances of the following 32-byte structure:

uint32              value_label_start;
uint32              value_label_end;
uint32              var_label_ofs;
uint32              format;
char                name[8];
union {
    flt64           f;
    char            s[8];
} missing;

The number of instances is the nominal_case_size specified in the main header record. There is one instance for each numeric variable and each string variable with width 8 bytes or less. String variables wider than 8 bytes have one instance for each 8 bytes, rounding up. The first instance for a long string specifies the variable’s correct dictionary information. Subsequent instances for a long string are generally filled with all-zero bytes, although the missing field contains the numeric system-missing value, and some writers also fill in var_label_ofs, format, and name, sometimes filling the latter with the numeric system-missing value rather than a text string. Regardless of the values used, readers should ignore the contents of these additional instances for long strings.

uint32 value_label_start;

uint32 value_label_end;

For a variable with value labels, these specify offsets into the label record of the start and end of this variable’s value labels, respectively. See Record 2: Labels Record, for more information.

For a variable without any value labels, these are both zero.

A long string variable may not have value labels.

uint32 var_label_ofs;

For a variable with a variable label, this specifies an offset into the label record. See Record 2: Labels Record, for more information.

For a variable without a variable label, this is zero.

uint32 format;

The variable’s output format, in the same format used in system files. See System File Output Formats, for details. SPSS/PC+ system files only use format types 5 (F, for numeric variables) and 1 (A, for string variables).

char name[8];

The variable’s name, padded on the right with spaces.

union { … } missing;

A user-missing value. For numeric variables, missing.f is the variable’s user-missing value. For string variables, missing.s is a string missing value. A variable without a user-missing value is indicated with missing.f set to the system-missing value, even for string variables (!). A Long string variable may not have a missing value.

In addition to the user-defined variables, every SPSS/PC+ system file contains, as its first three variables, the following system-defined variables, in the following order. The system-defined variables have no variable label, value labels, or missing values.

$CASENUM

A numeric variable with format F8.0. Most of the time this is a sequence number, starting with 1 for the first case and counting up for each subsequent case. Some files skip over values, which probably reflects cases that were deleted.

$DATE

A string variable with format A8. Same format (including varying padding) as the creation_date field in the main header record (see Record 0: Main Header Record). The actual date can differ from creation_date and from record to record. This may reflect when individual cases were added or updated.

$WEIGHT

A numeric variable with format F8.2. This represents the case’s weight; SPSS/PC+ files do not have a user-defined weighting variable. If weighting has not been enabled, every case has value 1.0.

6.3 Record 2: Labels Record ¶

The labels record holds value labels and variable labels. Unlike the other records, it is not meant to be read directly and sequentially. Instead, this record must be interpreted one piece at a time, by following pointers from the variables record.

The value_label_start, value_label_end, and var_label_ofs fields in a variable record are all offsets relative to the beginning of the labels record, with an additional 7-byte offset. That is, if the labels record starts at byte offset labels_ofs and a variable has a given var_label_ofs, then the variable label begins at byte offset labels_ofs + var_label_ofs + 7 in the file.

A variable label, starting at the offset indicated by var_label_ofs, consists of a one-byte length followed by the specified number of bytes of the variable label string, like this:

uint8               length;
char                s[length];

A set of value labels, extending from value_label_start to value_label_end (exclusive), consists of a numeric or string value followed by a string in the format just described. String values are padded on the right with spaces to fill the 8-byte field, like this:

union {
    flt64           f;
    char            s[8];
} value;
uint8               length;
char                s[length];

The labels record begins with a pair of uint32 values. The first of these is always 3. The second is between 8 and 16 less than the number of bytes in the record. Neither value is important for interpreting the file.

6.4 Record 3: Data Record ¶

The format of the data record varies depending on the value of compressed in the file header record:

0: no compression

Data is arranged as a series of 8-byte elements, one per variable instance variable in the variable record (see Record 1: Variables Record). Numeric values are given in flt64 format; string values are literal characters string, padded on the right with spaces when necessary to fill out 8-byte units.

1: bytecode compression

The first 8 bytes of the data record is divided into a series of 1-byte command codes. These codes have meanings as described below:

0

The system-missing value.

1

A numeric or string value that is not compressible. The value is stored in the 8 bytes following the current block of command bytes. If this value appears twice in a block of command bytes, then it indicates the second group of 8 bytes following the command bytes, and so on.

2 through 255

A number with value code - 100, where code is the value of the compression code. For example, code 105 indicates a numeric variable of value 5.

The end of the 8-byte group of bytecodes is followed by any 8-byte blocks of non-compressible values indicated by code 1. After that follows another 8-byte group of bytecodes, then those bytecodes’ non-compressible values. The pattern repeats up to the number of cases specified by the main header record have been seen.

The corpus does not contain any files with command codes 2 through 95, so it is possible that some of these codes are used for special purposes.

Cases of data often, but not always, fill the entire data record. Readers should stop reading after the number of cases specified in the main header record. Otherwise, readers may try to interpret garbage following the data as additional cases.

6.5 Records 4 and 5: Data Entry ¶

Records 4 and 5 appear to be related to SPSS/PC+ Data Entry.