Writing to files using buffers in Zig

In this post we’re going to go through four steps:

1. Create a new file to write into.
2. Construct a buffer to temporarily hold our bytes in memory.
3. Get our writer interface.
4. Write bytes to the file via the writer interface.

There’s working code at the end of this post if you want to skip to that.

Step 1: create a file

#

const std = @import("std");

pub fn main() !void {
  const file = try std.fs.cwd().createFile("example.txt", .{});
  defer file.close();
}

Construct a buffer

#

Next we need a buffer to hold our bytes before we send it to our destination. Buffers can take different shapes but it’s main purpose is managing overhead. Each time we write to a destination it requires a syscall (ask the operating system kernel to do something, usually on some hardware). Syscalls can be expensive because each time your application needs to switch context, make the request, wait for the response, then switch back. Each time you do this it’s nanoseconds but if you do this enough time it adds up fast.

To manage this we use buffers. We hold onto our bytes until the buffer fills then flush it to our destination, so we make fewer syscalls. The balance is here is that the bytes need to live somewhere in memory (usually) so we consume that resource in the name of performance. Fewer syscalls but more memory. It’s a balance.

In this post I’ll show two examples, a buffer on the stack, and a buffer on the heap. In this post I’m going with the stack buffer but I want to show examples of both and two rules of thumb on how to choose when to use them.

Stack buffer

#

A buffer on the stack has a compile-time known size so we need to decide how big our buffer will be. Auto-growing buffers isn’t a good idea here and not something built into Zig’s std library because of one of its rules, no hidden allocations.

const std = @import("std");

pub fn main() !void {
  const file = try std.fs.cwd().createFile("example.txt", .{});
  defer file.close();

+ var buf: [2048]u8 = undefined;
}

Heap buffer

#

A buffer on the heap has an unknown size at compile-time and can auto-grow to fit your needs with the constraints being the amount of available resources on the machine your code is running on. We’re using a GeneralPurposeAllocator¹ for learning purposes but feel free to use whatever allocation strategy suits your need.

const std = @import("std");

pub fn main() !void {
  const file = try std.fs.cwd().createFile("example.txt", .{});
  defer file.close();

+ var gpa = std.heap.GeneralPurposeAllocator(.{}){};
+ defer _ = gpa.deinit();
+
+ const allocator = gpa.allocator();
+
+ const buf = try allocator.alloc(u8, 2048);
+ defer allocator.free(buf);
}

Choosing between stack and heap buffers

#

If you know what you need your buffer size to be at compile-time or up to 16kb is enough for your use-case, then stack buffer is simpler and faster. If you don’t know how much buffer size you need or you know you’ll need more than 16kb, then buffer on the heap is likely your best bet. This is a general rule, no need to be rigid here, tune as you need.

We’ll continue with a stack buffer.

Get a writer interface

#

Let’s get our writer interface. Here, the caller owns the buffer and the writer borrows the buffer. To get our interface we first need to pass a pointer to our buffer, to our writer. This type of interface is called an intrusive interface, we’ll go over that in a different blog post.

const std = @import("std");

pub fn main() !void {
  const file = try std.fs.cwd().createFile("example.txt", .{});
  defer file.close();

  var buf: [2048]u8 = undefined;

  var file_writer = file.writer(&buf);
  const writer_iface: *std.Io.Writer = &file_writer.interface;
}

Write bytes to the file

#

When writing, bytes are copied into the buffer until it fills, at which point the buffer flushes to the file via the interface and the buffer pointer resets to the 0 index. Writing continues as many times as needed and a final flush will write any remaining partially filled buffer to the file. This is why you don’t forget to flush².

const std = @import("std");

pub fn main() !void {
  const file = try std.fs.cwd().createFile("example.txt", .{});
  defer file.close();

  var buf: [2048]u8 = undefined;

  var file_writer = file.writer(&buf);
  const writer_iface: *std.Io.Writer = &file_writer.interface;

  try writer_iface.writeAll("this is some slice of bytes to store in a file\n");
  try writer_iface.flush();
}

Our final working code compiles and writes this is some slice of bytes to store in a file to the file example.txt in your current working directory.