Introducing the `lofi` package for low-fidelity/low-bit data representations

2019-08-06

lofi

The goal of lofi is to squeeze multiple, low-fidelity representations of colours and numbers into the 32-bits of a single, standard integer in R.

This low-fidelity representation of values (a.k.a. lofi) is usually only an approximation of the original values, and reconstructed values will most likely be slightly different from the original.

Visit the webpage for more in-depth documentation.

What’s in the box

  • pack() and unpack() are the key functions for packing/unpacking multiple values into the bits of an integer
  • There is a suite of low-level functions for handling each particular supported type e.g. dbl_to_lofi() and lofi_to_dbl()

Installation

You can install from GitHub with:

# install.packages("devtools")
devtools::install_github("coolbutuseless/lofi")

pack() and unpack()

The pack() function will encode a set of values into the bits of a single integer. The unpack() function will reconstruct the original values from this single integer.

The schematic below illustrates the method by which a list of values is packed into a single integer and then unpacked back into the list of values. pack() and unpack() in turn rely on some lower level functions X_to_lofi() and lofi_to_X() (as indicated in the yellow boxes).

The key to the process is the packing specification (a.k.a. pack_spec) which defines the type of value you want to store how many bits are used to store each value.

Steps for using this package:

  1. Define a pack_spec defining how each value is converted to lofi
  2. Call pack() on a named list of values, along with the pack_spec
  3. Receive back a single integer containing all the packed lofi bits
  4. Call unpack() on the single integer (along with the pack_spec)
  5. Receive a list of reconstructed values which are the same as the original values - but maybe with some loss of precision.

The pack_spec defines information on packing the following types:

type lossless? nbits description signed
integer Yes 1-32 pack standard integer optional
logical Yes 1-32 standard 1-bit logical. zero-padded if needed NA
choice Yes 1-32 almost like a factor representation NA
double No 1-32 pack a standard double optional
colour No 3-24 pack a hex colour e.g. #123456 NA
scaled No 1-32 pack a range into the given bits NA
custom Possibly 1-32 user specified functions used to pack/unpack NA

The integer, logical and choice types are lossless, and original values can be perfectly reconstructed by unpack(). The double, colour and scaled types all quantize the inputs in some way and lose information - thus the original value is always imperfectly reconstructed (except in very particular circumstances).

For information on the specification for each type, see ?lofi::pack or vignette("packing-specification", package='lofi'

Example - pack/unpack the first row of iris data

The iris dataset gives the measurements in cm of the variables sepal length and width, and petal length and width, respectively, for 50 flowers from each of 3 species of iris. The first rows of the data are shown below:

Table 1: First rows of iris data
Sepal.Length Sepal.Width Petal.Length Petal.Width Species
5.1 3.5 1.4 0.2 setosa
4.9 3.0 1.4 0.2 setosa
4.7 3.2 1.3 0.2 setosa

The pack_spec for the data seen in iris is:

  • Sepal.Length is a floating point value with 1 decimal place with a maximum value of 7.9. This could be multiplied by 10, converted to an integer and stored in 7 bits.
  • Similarly for Sepal.Width, Petal.Length and Petal.Width - after multiplying by 10, and treating as an integer, these values could all by stored in 6, 7, and 5 bits respectively.
  • Species is a choice from 3 options, so in the best case we only need 2 bits to store this information.

The defined pack_spec is stored as a list:

#~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
# Can perfectly pack 'iris' into 27 bits per row.
#~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
pack_spec <- list(
  Sepal.Length = list(type = 'integer', nbits = 7, mult = 10, signed = FALSE),
  Sepal.Width  = list(type = 'integer', nbits = 6, mult = 10, signed = FALSE),
  Petal.Length = list(type = 'integer', nbits = 7, mult = 10, signed = FALSE),
  Petal.Width  = list(type = 'integer', nbits = 5, mult = 10, signed = FALSE),
  Species      = list(type = 'choice' , nbits = 2,
                      options = c('setosa', 'versicolor', 'virginica'))
)

Now take the first row of iris and pack() it:

lofi::pack(iris[1, ], pack_spec)
[1] 54052616

So the first row of iris has now been packed into the integer: 54052616.
If this integer is viewed as the 32 bits which make it up, the different lofi data representations can be identified:

If the integer is now unpack()ed, we get back the original data.

lofi::unpack(54052616L, pack_spec)
$Sepal.Length
[1] 5.1

$Sepal.Width
[1] 3.5

$Petal.Length
[1] 1.4

$Petal.Width
[1] 0.2

$Species
[1] "setosa"

For an example of using lofi to pack an entire data.frame see vignette("packing-a-data-frame", package='lofi')