serde_luaq

[!NOTE] This library is still a work in progress, and there are no API stability guarantees.

serde_luaq is a library for deserialising (and eventually, serialising) simple, JSON-equivalent data structures from Lua source code, without requiring Lua itself (unlike mlua).

The goal is to be able to read state from software (mostly games) which is serialised using Lua %q formatting (and similar techniques) without requiring arbitrary code execution.

This library consists of four parts:

• A LuaValue enum, which describes Lua's basic data types (nil, boolean, string, number, table).

• A peg-based parser for parsing a &[u8] (containing Lua) into a LuaValue.

• A serde-based Deserialize implementation for converting a LuaValue into your own data types.

• Optional lossy converter to and from serde_json's Value type.

Goal

For example, you could have a Lua script like this:

a = 1
b = {1, 2, 3}
c = {
    ["foo"] = "bar",
}

And define some Serde traits that let you deserialise it:

#[derive(Deserialize, PartialEq, Debug)]
struct ComplexType {
    foo: String,
}

#[derive(Deserialize, PartialEq, Debug)]
struct Test {
    a: u32,
    b: Vec<u32>,
    c: ComplexType,
}

Then deserialise it with:

use serde_luaq::{from_slice, LuaFormat};

let parsed: Test = serde_luaq::from_slice(input, LuaFormat::Script).unwrap();

assert_eq!(parsed, Test {
  a: true,
  b: vec![1, 2, 3],
  c: ComplexType { foo: "bar".to_string() },
});

Parser features

• Input formats
  • Bare Lua value expression, similar to JSON ({["hello"] = "world"})
  • Lua return statement (return {["hello"] = "world"})
  • Script with identifier assignments only (hello = "world")
• Serde (partial)
  • Deserialising
  • Serialising
• Lossy serde_json interoperability
  • LuaValue -> serde_json::Value
  • serde_json::Value -> LuaValue

Lua language features

This library aims to implement a subset of Lua that is equivalent to the subset of JavaScript that a JSON parser would implement:

• nil
• Booleans (true, false)
• Numbers
  • Integers
    • Decimal integers (123)
      • Coercion to float for decimal numbers < i64::MIN or > i64::MAX
    • Hexadecimal integer (0xFF)
      • Wrapping large hexadecimal numbers to i64
  • Floats
    • Decimal floats with decimal point and optional exponent (3.14, 0.314e1)
    • Decimal floats with mandatory exponent (3e14)
    • Hexadecimal floating points (0x.ABCDEFp+24)
    • Positive and negative infinity (1e9999, -1e9999)
    • NaN ((0/0))
• Strings
  • Strings in single quotes (')
  • Strings in double quotes (")
  • Strings in long brackets ([[string]], [==[string]==]) (up to 5 = deep)
  • Arbitrary 8-bit binary data inside strings (like [u8])
  • Escapes in quoted strings:
    • C-like backslash-escapes (abfnrtv\"')
    • Escaped line breaks (\\\n, \\\r, \\\r\n, \\\n\r)
    • \z whitespace span escapes (str\z ing == string)
    • Decimal escape sequences (\1, \01, \001)
    • Hexadecimal escape sequences (\x01)
    • UTF-8 escape sequences (\u{1F4A9})
      • Sequences allowed by RFC 2279 but not RFC 3629 (\u{D800}, \u{7FFFFFFF})
• Tables
  • Key-values / expression keys ({["foo"] = "bar"}, {[1234]="bar"})
  • Name-values / identifier keys ({foo = "bar"})
  • Values / implicit keys ({"bar"})
  • Mixed key types
  • Identifier validation

This library is not designed to replace Lua, nor execute arbitrary Lua code, so these Lua features are intentionally unsupported:

• Arithmetic operators (+, -, *, /...)
• Bitwise operators (<<, >>, &, |, ~...)
• Blocks and visibility modifiers (do ... end, local)
• Comments
• Control structures (if, break, for, goto, repeat, until, while...)
• Function calls
• Function definitions
• Length operator (#)
• Logical operators (and, or, not)
• Newline character normalisation in strings (\r\n => \n on UNIX, \n => \r\n on Windows)
• Parentheses, except for (0/0) (NaN)
• Pointers (light userdata)
• Referencing other variables (a = 10; b = a)
• Relational operators (==, ~=, <, >...)
• String concatenation ("hello" .. " world")
• Threads and coroutines
• Updating other variables (a = {}; a.b = 'foo')
• Userdata
• Vararg assignments and destructuring (a, b = 1, 2)

If you want to use these language features or otherwise need to run arbitrary Lua code, look at something like mlua, which links to liblua, and also provides serde bindings.

Known users of Lua serialisation

• SaveData: Love2D library, emits a return statement which is loaded by evaluating the string.

• Balatro (engine/string_packer.lua) is a modified version of SaveData that also compresses with deflate for save games and settings (.jkr files).

• World of Warcraft: addon state, written to WTF/{Account,SavedVariables}/**/*.lua as scripts that set variables.