The final chapter. We tackle the most conceptually challenging feature: capturing variables from enclosing scopes.
The problem #
Consider this Monkey program:
let newAdder = fn(a) {
fn(b) { a + b; };
};
let addTwo = newAdder(2);
addTwo(3); // => 5
When addTwo(3) executes, the parameter a from newAdder is no longer on
the stack – newAdder has already returned. The inner function needs to
capture (“close over”) a at the time it is created. That captured binding is
called a free variable, and the function together with its captured
bindings is called a closure.
AST and parser changes #
Recursive closures need to know the function’s name (from let countDown = fn(x) {...}). Add a name field to FunctionLiteral in ast.zig:
pub const FunctionLiteral = struct {
parameters: []const Identifier,
body: BlockStatement,
name: []const u8 = "",
};
The default "" means anonymous functions (not bound via let) work unchanged.
Update parseLetStatement in parser.zig to set the name when the value is a function literal. Change const value to var value and add the check after parsing:
fn parseLetStatement(self: *Parser) Error!?ast.Statement {
if (!try self.expectPeek(.ident)) return null;
const name = self.cur_token.literal;
if (!try self.expectPeek(.assign)) return null;
self.nextToken();
var value = try self.parseExpression(.lowest) orelse return null;
if (value == .function_literal) value.function_literal.name = name;
if (self.peek_token.type == .semicolon) self.nextToken();
return .{ .let_statement = .{ .name = name, .value = value } };
}
New object: Closure
#
Add this struct to object.zig:
pub const Closure = struct {
fn_obj: CompiledFunction, // The underlying compiled function.
free: []const Object, // Captured free variables, in the order they were resolved.
};
All functions become closures. A plain function is just a closure with an
empty free slice. This simplifies the VM because there is only one callable type
to handle (besides builtins).
Add a closure variant to the existing Object union in object.zig:
pub const Object = union(enum) {
integer: i64,
boolean: bool,
string: []const u8,
array: []const Object,
hash: HashPair,
null,
compiled_function: CompiledFunction,
closure: Closure,
builtin: Builtin,
};
The compiled_function variant stays in the union because constants still
store raw CompiledFunction objects. The op_closure instruction wraps them
into Closure at runtime.
New opcodes #
| Opcode | Operands | Description |
|---|---|---|
op_closure |
u16 (const idx) + u8 (count) |
Create a Closure from a CompiledFunction constant |
op_get_free |
u8 (index) |
Push current_closure.free[index] |
op_current_closure |
(none) | Push the current frame’s closure onto the stack |
Add these variants to the Opcode enum in code.zig:
op_closure,
op_get_free,
op_current_closure,
Add these entries to the lookup function in code.zig:
.op_closure => .{ .name = "OpClosure", .operand_widths = &[_]u8{ 2, 1 } },
.op_get_free => .{ .name = "OpGetFree", .operand_widths = &[_]u8{1} },
.op_current_closure => .{ .name = "OpCurrentClosure", .operand_widths = &[_]u8{} },
Symbol table changes #
All code in this section goes in symbol_table.zig.
Two new scopes #
Add two new variants to the SymbolScope enum. After chapter 12, your enum has global, local, and builtin. Add free and function:
pub const SymbolScope = enum {
global,
local,
builtin,
free,
function,
};
free– a variable captured from an enclosing scope.function– the function’s own name (for recursion).
Tracking free symbols #
Add a free_symbols field to SymbolTable. This tracks which symbols from outer scopes were captured as free variables during resolution. Update the struct to include this new field and update init and initEnclosed accordingly:
pub const SymbolTable = struct {
allocator: std.mem.Allocator,
outer: ?*SymbolTable,
store: std.StringHashMap(Symbol),
num_definitions: usize,
/// Free variables discovered during resolution in this scope.
free_symbols: std.ArrayList(Symbol),
pub fn init(allocator: std.mem.Allocator) SymbolTable {
return .{
.allocator = allocator,
.outer = null,
.store = std.StringHashMap(Symbol).init(allocator),
.num_definitions = 0,
.free_symbols = .empty,
};
}
pub fn initEnclosed(allocator: std.mem.Allocator, outer: *SymbolTable) SymbolTable {
var st = init(allocator);
st.outer = outer;
return st;
}
pub fn deinit(self: *SymbolTable) void {
self.store.deinit();
self.free_symbols.deinit(self.allocator);
}
// ...
};
Defining a free symbol #
Add the defineFree method to SymbolTable. When resolve finds a name in an outer scope that is neither global nor builtin, it promotes the symbol to a free variable in the current scope:
/// Record `original` (from an outer scope) as a free variable in this
/// scope. Returns a new Symbol with `.free` scope whose index is the
/// position in `free_symbols`.
pub fn defineFree(self: *SymbolTable, original: Symbol) !Symbol {
try self.free_symbols.append(self.allocator, original);
const symbol = Symbol{
.name = original.name,
.scope = .free,
.index = self.free_symbols.items.len - 1,
};
try self.store.put(original.name, symbol);
return symbol;
}
Defining the function name #
Add the defineFunctionName method to SymbolTable. For recursive closures, the function’s own name resolves to function scope:
/// Define the function's own name so it can reference itself recursively.
/// This does NOT increment `num_definitions` -- the function does not
/// occupy a local slot.
pub fn defineFunctionName(self: *SymbolTable, name: []const u8) !Symbol {
const symbol = Symbol{
.name = name,
.scope = .function,
.index = 0,
};
try self.store.put(name, symbol);
return symbol;
}
Updated resolve
#
Replace the resolve method on SymbolTable. The previous version (from chapter 11) simply walked the outer chain. The new version adds free variable promotion: when a symbol is found in an outer scope as .local or .free, it is promoted to a .free variable in the current scope:
pub fn resolve(self: *SymbolTable, name: []const u8) ?Symbol {
// Check the current scope's store.
if (self.store.get(name)) |sym| return sym;
// If there is no outer scope, the name is undefined.
const outer = self.outer orelse return null;
// Resolve in the outer scope.
const obj = outer.resolve(name) orelse return null;
// Global and builtin symbols pass through unchanged.
if (obj.scope == .global or obj.scope == .builtin) return obj;
// Local or free symbols from the outer scope become free variables in the current scope.
const free = self.defineFree(obj) catch return null;
return free;
}
This is recursive: if scope C references a variable defined in scope A (two levels up), then:
- C’s
resolvecalls B’sresolve - B’s
resolvefinds it in A as.local - B promotes it to
.freein B - C sees it as
.freein B and promotes it to.freein C
Each level in the chain captures the variable from the level above.
Compiler changes #
All code in this section goes in compiler.zig.
Updated deinit
#
Update the deinit method on the Compiler struct to also clean up the symbol table’s free_symbols:
pub fn deinit(self: *Compiler) void {
for (self.scopes.items) |*scope| {
scope.instructions.deinit(self.allocator);
}
self.scopes.deinit(self.allocator);
self.constants.deinit(self.allocator);
self.symbol_table.deinit();
}
Replace OpConstant with OpClosure for functions
#
Replace the .function_literal case in compileExpression. Every function literal now emits op_closure instead of op_constant. The compiler also emits load instructions for each free variable before the op_closure instruction:
.function_literal => |fl| {
try self.enterScope();
// If the function has a name (from a let statement), define it
// so recursive references resolve to `function` scope.
if (fl.name.len > 0) _ = try self.symbol_table.defineFunctionName(fl.name);
// Define parameters as locals.
for (fl.parameters) |param| _ = try self.symbol_table.define(param.value);
// Compile the body.
for (fl.body.statements) |stmt| try self.compileStatement(stmt);
// Handle implicit/explicit returns.
if (self.lastInstructionIs(.op_pop)) self.replaceLastPopWithReturn();
if (!self.lastInstructionIs(.op_return_value)) _ = try self.emit(.op_return, &[_]usize{});
// Capture free symbols before leaving scope
// (leaveScope destroys the current symbol table).
const free_symbols = try self.allocator.dupe(
Symbol,
self.symbol_table.free_symbols.items,
);
const num_locals = self.symbol_table.num_definitions;
const instructions = try self.leaveScope();
// Emit load instructions for each free variable. These push the
// captured values onto the stack so OpClosure can scoop them up.
for (free_symbols) |sym| try self.loadSymbol(sym);
const compiled_fn = Object{ .compiled_function = .{
.instructions = instructions,
.num_locals = num_locals,
.num_parameters = fl.parameters.len,
} };
const const_idx = try self.addConstant(compiled_fn);
// OpClosure: constant index (u16) + number of free variables (u8)
_ = try self.emit(.op_closure, &[_]usize{ const_idx, free_symbols.len });
},
Updated loadSymbol
#
Replace loadSymbol in compiler.zig. Add the new .free and .function cases to the existing .global, .local, and .builtin cases:
fn loadSymbol(self: *Compiler, sym: Symbol) CompileError!void {
switch (sym.scope) {
.global => _ = try self.emit(.op_get_global, &[_]usize{sym.index}),
.local => _ = try self.emit(.op_get_local, &[_]usize{sym.index}),
.builtin => _ = try self.emit(.op_get_builtin, &[_]usize{sym.index}),
.free => _ = try self.emit(.op_get_free, &[_]usize{sym.index}),
.function => _ = try self.emit(.op_current_closure, &[_]usize{}),
}
}
VM changes #
All code in this section goes in vm.zig.
Add Closure to the existing imports at the top of vm.zig:
const Closure = object.Closure;
(CompiledFunction and Object should already be imported from chapter 11.)
Frames hold closures #
Update the Frame struct in vm.zig. Replace the fn_obj: CompiledFunction field with cl: Closure:
pub const Frame = struct {
cl: Closure, // The closure being executed.
ip: usize, // Instruction pointer within this frame's instructions.
base_pointer: usize, // Stack index where this frame's locals begin.
pub fn instructions(self: *const Frame) []const u8 {
return self.cl.fn_obj.instructions;
}
};
Update VM.init in vm.zig. The method already takes allocator as its first parameter (from chapter 10). Replace the previous frame initialization that used a bare CompiledFunction with this version that wraps the main program in a Closure:
const main_fn = CompiledFunction{
.instructions = bytecode.instructions,
.num_locals = 0,
.num_parameters = 0,
};
const main_closure = Closure{
.fn_obj = main_fn,
.free = &[_]Object{},
};
const main_frame = Frame{
.cl = main_closure,
.ip = 0,
.base_pointer = 0,
};
vm.pushFrame(main_frame);
Executing op_closure
#
Add this case to switch (op) in the run method:
.op_closure => {
// Read the constant index (u16, big-endian).
const const_index = std.mem.readInt(
u16,
ins[ip + 1 ..][0..2],
.big,
);
const num_free: usize = @intCast(ins[ip + 3]);
frame.ip += 4;
const compiled_fn = self.constants[const_index].compiled_function;
// Pop the free variables from the stack. They were pushed in order
// by the compiler, so they sit at sp -> num_free .. sp.
const free = try self.allocator.alloc(Object, num_free);
for (0..num_free) |i| free[i] = self.stack[self.sp - num_free + i].?;
self.sp -= num_free;
const closure = Closure{
.fn_obj = compiled_fn,
.free = free,
};
try self.push(Object{ .closure = closure });
continue;
},
Executing op_get_free
#
Add this case to switch (op) in the run method:
.op_get_free => {
const free_index: usize = @intCast(ins[ip + 1]);
frame.ip += 2;
const current_closure = self.currentFrame().cl;
try self.push(current_closure.free[free_index]);
continue;
},
Executing op_current_closure
#
Add this case to switch (op) in the run method:
.op_current_closure => {
frame.ip += 1;
const current_closure = self.currentFrame().cl;
try self.push(Object{ .closure = current_closure });
continue;
},
Updated op_call for closures
#
Replace the .op_call case in the run method. It now handles .closure instead of .compiled_function:
.op_call => {
const num_args: usize = @intCast(ins[ip + 1]);
frame.ip += 2;
const callee = self.stack[self.sp - 1 - num_args].?;
switch (callee) {
.closure => |cl| {
if (num_args != cl.fn_obj.num_parameters) {
return error.WrongArgumentCount;
}
const new_frame = Frame{
.cl = cl,
.ip = 0,
.base_pointer = self.sp - num_args,
};
self.pushFrame(new_frame);
self.sp = new_frame.base_pointer + cl.fn_obj.num_locals;
continue;
},
.builtin => |b| {
var args_buf: [256]Object = undefined;
for (0..num_args) |i| {
args_buf[i] = self.stack[self.sp - num_args + i].?;
}
const args = args_buf[0..num_args];
const result = b.func(self.allocator, args);
self.sp = self.sp - num_args - 1;
try self.push(result);
continue;
},
else => return error.CallingNonFunction,
}
},
How closures work end to end #
Walk through this example:
let newAdder = fn(a) {
fn(b) { a + b; };
};
let addTwo = newAdder(2);
addTwo(3); // => 5
Compilation #
-
Compile the inner function
fn(b) { a + b; }.bresolves as local (index 0).ais not local.resolvechecks the outer scope and findsaas local (index 0) innewAdder’s scope. It promotesato.free(index 0) in the inner scope.- The inner function’s body emits:
OpGetFree 0,OpGetLocal 0,OpAdd,OpReturnValue. free_symbolsfor the inner scope is[Symbol{ .name = "a", .scope = .local, .index = 0 }].
-
After leaving the inner scope, the compiler emits:
OpGetLocal 0– loadafromnewAdder’s locals (the free symbol’s original location).OpClosure <const_idx> 1create a closure with 1 free variable.
-
Compile
newAdder. Its body produces the instructions from step 2. ThennewAdderitself is wrapped in anOpClosurewith 0 free variables.
Execution #
-
newAdder(2): pushes 2 as argument, callsnewAdder.- Inside
newAdder,ais local 0 with value 2. OpGetLocal 0pushes 2 onto the stack.OpClosure <idx> 1pops 1 value (2) from the stack and createsClosure{ .fn_obj = inner, .free = [2] }.OpReturnValuereturns this closure.
- Inside
-
addTwo(3): pushes 3 as argument, calls the closure.- Inside the closure,
bis local 0 with value 3. OpGetFree 0pushesclosure.free[0]= 2.OpGetLocal 0pushes 3.OpAddproduces 5.
- Inside the closure,
Nested closures #
For deeper nesting, the free-variable chain propagates through each level:
let newAdderOuter = fn(a) {
fn(b) {
fn(c) { a + b + c; };
};
};
- The innermost function has
aandbas free variables. - The middle function has
aas a free variable (from the outer scope). - When compiling the middle function:
bis local (index 0).ais free (index 0), captured fromnewAdderOuter.
- When compiling the innermost function:
cis local (index 0).bresolves in the middle scope as local 0 and becomes free 0 in the innermost scope.aresolves in the middle scope as free 0 and becomes free 1 in the innermost scope.
The compiler emits for the innermost closure:
OpGetLocal 0 // load b from middle scope
OpGetFree 0 // load a from middle scope's free[0]
OpClosure <idx> 2
And the VM captures both values when creating the closure.
Recursive closures #
A function that calls itself by name needs to reference its own closure:
let countDown = fn(x) {
if (x == 0) {
return 0;
} else {
countDown(x - 1);
}
};
countDown(1); // => 0
compilation #
When compiling the function literal bound to countDown:
enterScopeis called.defineFunctionName("countDown")stores a symbol with.functionscope.- Inside the body,
countDownresolves to.functionscope. loadSymbolemitsOpCurrentClosure.
The emitted bytecode for the recursive call:
OpCurrentClosure // push the closure itself
OpGetLocal 0 // push x
OpConstant 1 // push 1
OpSub // x - 1
OpCall 1 // call countDown(x - 1)
execution #
OpCurrentClosure pushes self.currentFrame().cl (the closure currently
being executed) onto the stack. This is then used as the callee by OpCall.
Why not free variables? #
You might think the function name could be a free variable. But the closure
does not exist yet when we would need to capture it. OpCurrentClosure
sidesteps this by reading from the current frame at runtime.
Tests #
Symbol table tests #
Add these tests inline in symbol_table.zig.
test "resolve free" {
var global = SymbolTable.init(testing.allocator);
defer global.deinit();
_ = try global.define("a");
_ = try global.define("b");
var first_local = SymbolTable.initEnclosed(testing.allocator, &global);
defer first_local.deinit();
_ = try first_local.define("c");
_ = try first_local.define("d");
var second_local = SymbolTable.initEnclosed(testing.allocator, &first_local);
defer second_local.deinit();
_ = try second_local.define("e");
_ = try second_local.define("f");
// From second_local:
// a, b => global (pass through)
// c, d => free (captured from first_local)
// e, f => local
const resolved_a = second_local.resolve("a").?;
try testing.expectEqual(SymbolScope.global, resolved_a.scope);
const resolved_c = second_local.resolve("c").?;
try testing.expectEqual(SymbolScope.free, resolved_c.scope);
try testing.expectEqual(@as(usize, 0), resolved_c.index);
const resolved_d = second_local.resolve("d").?;
try testing.expectEqual(SymbolScope.free, resolved_d.scope);
try testing.expectEqual(@as(usize, 1), resolved_d.index);
const resolved_e = second_local.resolve("e").?;
try testing.expectEqual(SymbolScope.local, resolved_e.scope);
try testing.expectEqual(@as(usize, 0), resolved_e.index);
// Verify that first_local's free_symbols is empty (c, d are local there).
try testing.expectEqual(@as(usize, 0), first_local.free_symbols.items.len);
// Verify that second_local's free_symbols has c and d.
try testing.expectEqual(@as(usize, 2), second_local.free_symbols.items.len);
try testing.expectEqualStrings("c", second_local.free_symbols.items[0].name);
try testing.expectEqualStrings("d", second_local.free_symbols.items[1].name);
}
test "resolve function name" {
var global = SymbolTable.init(testing.allocator);
defer global.deinit();
var local = SymbolTable.initEnclosed(testing.allocator, &global);
defer local.deinit();
_ = try local.defineFunctionName("myFunc");
const resolved = local.resolve("myFunc").?;
try testing.expectEqual(SymbolScope.function, resolved.scope);
try testing.expectEqual(@as(usize, 0), resolved.index);
}
Update the chapter 11 compiler test #
The “function literal with return” test from chapter 11 expected OpConstant for the function. Now that all functions are closures, update it to use OpClosure:
Change this line in compiler_test.zig:
// Old (chapter 11):
try code.make(allocator, .op_constant, &[_]usize{2}),
// New (chapter 13):
try code.make(allocator, .op_closure, &[_]usize{ 2, 0 }),
And update the comment above it from OpConstant 2 to OpClosure 2, 0.
Compiler tests #
Add these tests to compiler_test.zig.
test "compile closure" {
const allocator = std.testing.allocator;
const input = "fn(a) { fn(b) { a + b; }; }";
var arena = std.heap.ArenaAllocator.init(allocator);
defer arena.deinit();
var l = Lexer.init(input);
var p = Parser.init(arena.allocator(), &l);
const program = try p.parseProgram();
var comp = try Compiler.init(arena.allocator());
try comp.compile(program);
// Inspect the outer function constant to verify OpClosure is emitted.
const bytecode = comp.bytecode();
// The outermost expression is an OpClosure with 0 free vars.
// Detailed byte-level assertions depend on your exact constant layout.
try std.testing.expect(bytecode.instructions.len > 0);
}
VM tests #
Add these tests to vm_test.zig.
test "vm: closures" {
const allocator = std.testing.allocator;
const closure_vm_tests = [_]struct {
input: []const u8,
expected: ExpectedValue,
}{
// Basic closure.
.{
.input =
\\let newClosure = fn(a) {
\\ fn() { a; };
\\};
\\let closure = newClosure(99);
\\closure();
,
.expected = .{ .integer = 99 },
},
// Closure with computation.
.{
.input =
\\let newAdder = fn(a, b) {
\\ fn(c) { a + b + c; };
\\};
\\let adder = newAdder(1, 2);
\\adder(8);
,
.expected = .{ .integer = 11 },
},
// Nested closures.
.{
.input =
\\let newAdderOuter = fn(a) {
\\ fn(b) {
\\ fn(c) { a + b + c; };
\\ };
\\};
\\let newAdderInner = newAdderOuter(1);
\\let adder = newAdderInner(2);
\\adder(3);
,
.expected = .{ .integer = 6 },
},
// Closure capturing a closure.
.{
.input =
\\let a = 1;
\\let newAdderOuter = fn(b) {
\\ fn(c) {
\\ fn(d) { a + b + c + d; };
\\ };
\\};
\\let newAdderInner = newAdderOuter(2);
\\let adder = newAdderInner(3);
\\adder(8);
,
.expected = .{ .integer = 14 },
},
// Closure used multiple times.
.{
.input =
\\let newClosure = fn(a, b) {
\\ let one = fn() { a; };
\\ let two = fn() { b; };
\\ one() + two();
\\};
\\newClosure(9, 90);
,
.expected = .{ .integer = 99 },
},
// Recursive closure.
.{
.input =
\\let countDown = fn(x) {
\\ if (x == 0) {
\\ return 0;
\\ } else {
\\ countDown(x - 1);
\\ }
\\};
\\countDown(1);
,
.expected = .{ .integer = 0 },
},
// Recursive closure: wrapper pattern.
.{
.input =
\\let wrapper = fn() {
\\ let countDown = fn(x) {
\\ if (x == 0) {
\\ return 0;
\\ } else {
\\ countDown(x - 1);
\\ }
\\ };
\\ countDown(1);
\\};
\\wrapper();
,
.expected = .{ .integer = 0 },
},
// Fibonacci.
.{
.input =
\\let fibonacci = fn(x) {
\\ if (x == 0) {
\\ return 0;
\\ } else {
\\ if (x == 1) {
\\ return 1;
\\ } else {
\\ fibonacci(x - 1) + fibonacci(x - 2);
\\ }
\\ }
\\};
\\fibonacci(15);
,
.expected = .{ .integer = 610 },
},
};
for (closure_vm_tests) |tt| {
var arena = std.heap.ArenaAllocator.init(allocator);
defer arena.deinit();
const result = try testVMRun(&arena, tt.input);
try testExpectedValue(result, tt.expected);
}
}
Benchmark: VM vs. interpreter #
With closures complete, the entire Monkey language is now compiled. A good benchmark is recursive Fibonacci:
let fibonacci = fn(x) {
if (x == 0) {
return 0;
} else {
if (x == 1) {
return 1;
} else {
fibonacci(x - 1) + fibonacci(x - 2);
}
}
};
fibonacci(35);
Create a new file benchmark.zig. This is a standalone program for comparing VM vs interpreter performance:
const std = @import("std");
pub fn main() !void {
const allocator = std.heap.page_allocator;
const input =
\\let fibonacci = fn(x) {
\\ if (x == 0) { return 0; } else {
\\ if (x == 1) { return 1; } else {
\\ fibonacci(x - 1) + fibonacci(x - 2);
\\ }
\\ }
\\};
\\fibonacci(35);
;
// Interpreter.
{
const program = try parse(input);
var env = Environment.init(allocator);
defer env.deinit();
var timer = try std.time.Timer.start();
const result = try evaluate(program, &env);
const elapsed = timer.read();
std.debug.print(
"evaluator: result={} duration={d:.3}s\n",
.{ result.integer, @as(f64, @floatFromInt(elapsed)) / 1_000_000_000 },
);
}
// Compiler/VM.
{
const program = try parse(input);
var comp = try Compiler.init(allocator);
try comp.compile(program);
const bytecode = comp.bytecode();
var vm = VM.init(allocator, bytecode);
var timer = try std.time.Timer.start();
try vm.run();
const elapsed = timer.read();
const result = vm.lastPoppedStackElem().?;
std.debug.print(
"vm: result={} duration={d:.3}s\n",
.{ result.integer, @as(f64, @floatFromInt(elapsed)) / 1_000_000_000 },
);
}
}
Expected output (times vary by machine):
evaluator: result=9227465 duration=25.380s
vm: result=9227465 duration=8.120s
The VM should be roughly 3x faster than the tree-walking evaluator. The
speedup comes from no AST traversal overhead per operation, no environment
hash-map lookups for variable access (locals are stack indexed), no recursive
Eval calls (the VM loop is a flat switch), and better cache locality from
the contiguous bytecode stream.
Verify it works #
Run zig build test. No output means all tests passed.