Skip to content

VLQ inefficiencies and opportunities #259

Description

@rakudrama

There are two kinds of inefficiency in the VLQ encoding. Multiple VLQ codes decode to the same value, and some decoded values are erroneous in context. This redundancy might be usefully repurposed.

Duplicate encodings

A, gA, ggA, ... all encode 0.
C, iA, igA, ... all encode -1 (signed) or 2 (unsigned).

B, hA, ghA, ... all encode -231 (signed) or 1 (unsigned).
(Some decoders get this wrong as a delta of -231 is not useful so never appears in practice).

Erroneous values

The Mappings field uses signed VLQ values for the generated column. Many encodings are wasted, since they would cause GeneratedColumn to become negative.

What can we do with this?

Mixed absolute and relative encodings

The Scopes proposal encodes Scope start and end offsets into the original source as unsigned offsets.
Near the end of #73 (comment) is a suggestion how to 'go backwards' to create two scopes covering the same source element. An alternative to this extra element is to enrich VLQ encodings to allow the same position to be encoded. If a VLQ ends with A to add a useless high order VLQ digit zero, that is currently redundant. This slightly longer encoding could be used to encode an Absolute value rather than a relative one. The encoding A would still be a relative offset of zero.

Sometimes the index into the names list needs to go to a very low index. An absolute encoding would be better here than a large negative offset.

Encode nulls

The signed value -231 encoded by B will never occur in practice as a signed relative value.
Accessing names uses signed relative values. There are some string values that are optional. This could be encoded with B.
If there is ever a context where both null and -231 are valid, the latter could be encoded as hA.

Increased number of relative offsets

A single digit VLQ encodes a signed range of ±15, double digit ±511 etc.
If the signed offset causes the value to become negative, the negative value could be reinterpreted as extending the positive range.
Example: the current position is 0. ±15 has 15 useless values, They could encode +16 to +30.
Example: the current position is 4, the next is 20. +16 requires a 2-digit VLQ, but if we re-interpret the 11 useless values -15...-5 as +16...+26 then we need only one digit.

The above is compatible with the current encoding for mappings. It might be easier to reason about an incompatible scheme where the size of the VLQ in digits is used to determine whether a negative value is possible after applying any signed offset and instead interpret the value as a unsigned if the signed range has any useless encodings. This would be choosing between a signed and unsigned offset on the basis of whether a negative value is possible rather than achieved as an intermediate step. This is a function of the number of VLQ digits.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions