In which I try to figure out how to pack and unpack bytes over an in-process pipe so that I can use it in some future message framing protocol for a worker pool. There will be a guest appearance by Fiber to simplify the parsing of messages in a non-blocking manner.
Some of the pieces I will need are IO.pipe, IO.select, Array::pack, String::unpack, Fiber.new, and Fiber.resume. We’ll start with pack
irb(main):041:0> [512].pack("C*")
=> "\x00"
irb(main):042:0> [512].pack("S*")
=> "\x00\x02"
irb(main):044:0> [512].pack("C*")
=> "\x00"
irb(main):045:0> [512].pack("S*")
=> "\x00\x02"
irb(main):046:0> [512].pack("L*")
=> "\x00\x02\x00\x00"
irb(main):047:0> [512].pack("Q*")
=> "\x00\x02\x00\x00\x00\x00\x00\x00"
irb(main):048:0> [512].pack("n*")
=> "\x02\x00"
irb(main):049:0> [512].pack("N*")
=> "\x00\x00\x02\x00"
Seems sensible. It takes an array of bytes and “directives” describing how to treat the elements of the array and then returns a binary representation of the array. The representation is given to us as a string but for our purposes we can pretend it is a byte array.
It should be somewhat clear what unpack will do because it will be the inverse operation for pack. When writing the network protocol it is very important that the client and server agree on the directives used to encode the message because otherwise one side or the other will be confused about what the byte array means. I still don’t understand how big-endian vs little-endian works out here but I think as long as long as I’m consistent with what directive I use on each side I should be fine
irb(main):050:0> "\x00\x00\x02\x00".unpack("N*")
=> [512]
irb(main):051:0> "\x00\x00\x02\x00".unpack("n*")
=> [0, 512]
I have to make a choice here about the length of the prefix which in turn will force a maximum message size. I’m going to use 2 bytes for the message length which means the maximum message size will be 0xFFFF (65535 bytes ~ 64 kilobytes). Restricting the message size to 64 kb might seem like a significant restriction but it’s not. We can always implement a multi-part protocol on top of it at a small increase in complexity of the server and client parsers. From the above output it looks like the directive I need to produce 2 byte output is 'n'.
Onto the IO.pipe shenanigans. First, the general structure to simulate producer/consumer model with some threads
r, w = IO.pipe # writer writer = Thread.new do # ... end # reader reader = Thread.new do # ... end # block main thread on the workers [reader, writer].each(&:join)
Let’s fill in the writer thread. When sending the message I’m going to do it in 2 parts to simulate some network slowness to force the reader to deal with such issues gracefully
writer = Thread.new do
begin
loop do
l = rand(0xffff)
STDOUT.puts "Sending message length: #{l}"
packed_length = [l].pack('n')
# write the length of the message but be slow about it
packed_length.each_char {|b| sleep 1; w.write b}
# write the first half of the message
(l / 2).times {w.write [rand(0xff)].pack('C')}
# write the second half of the message after a delay
sleep 1
(l / 2).times {w.write [rand(0xff)].pack('C')}
# if the message length was odd then we need to write 1 more byte
w.write [rand(0xff)].pack('C') if l % 2 == 1
STDOUT.puts "Message sent"
end
rescue => e
STDERR.puts e
end
end
Now the reader. This is where Fiber will make an appearance. I highly recommend trying to write the reader without Fiber to see the pain points that Fiber solves
def make_reader
#...
end
reader = Thread.new do
reader_fiber = make_reader
loop do
# first we wait on the pipe to be readable
IO.select [r]
# now we pass it into the fiber for processing
reader_fiber.resume(r)
end
end
So the main reader loop just uses IO.select to wait on a pipe and when it is ready for reading passes it along to the fiber. Lets see what the fiber looks like
def make_reader
Fiber.new do |pipe|
loop do
STDOUT.puts "Resetting buffers"
l_buffer, m_buffer = "", ""
loop do
# per our protocol the first 2 bytes is the message length
to_read = 2 - l_buffer.length
# the buffer is the right size so time to parse the message
break if to_read.zero?
# buffer is not full yet so try to read the right number of bytes
begin
pipe.read_nonblock(to_read).each_char {|b| l_buffer << b}
rescue IO::WaitReadable
# we tried to read but pipe wasn't ready so wait to be resumed
Fiber.yield
end
end
# length buffer is full at this point so we can decode it
decoded_length = l_buffer.unpack('n').first
STDOUT.puts "Decoded length: #{decoded_length}"
# now we need to start accumulating the message bytes
loop do
# see how many more bytes we need to read
to_read = decoded_length - m_buffer.length
# looks like we have all the bytes
break if to_read.zero?
# we don't have all the bytes so try to read
begin
pipe.read_nonblock(to_read).each_char {|b| m_buffer << b}
rescue IO::WaitReadable
# we tried to read but not ready so we yield until there is data to read
Fiber.yield
end
end
# got all the message bytes so decode and act on it
STDOUT.puts "Received message buffer length: #{m_buffer.length}"
# time to parse some more messages
STDOUT.puts "Going to top of loop to parse more messages"
end
end
end
Seems like a lot but it is pretty simple. We just try to read in a non-blocking manner from the pipe that was passed to the fiber. If the pipe doesn’t have what we need then we try to continuously read in a non-blocking manner from the pipe, yielding to the main IO.select loop when the pipe is completely empty and trying to read as many bytes as possible when it is not empty. The pipe being empty is indicated by an exception (IO::WaitReadable) and in that case we just yield and wait to be resumed. I highly recommend trying to do the same thing without Fiber to see the value that Fiber adds.
As a skeleton this will work out as a nice playground for experimenting further. Here’s some output to demonstrate how the reader and writer interact
Sending message length: 4292 Resetting buffers Decoded length: 4292 Message sentReceived message buffer length: 4292 Going to top of loop to parse more messages Resetting buffers Sending message length: 50469 Decoded length: 50469 Message sent Received message buffer length: 50469 Going to top of loop to parse more messages Resetting buffers Sending message length: 59611 Decoded length: 59611 Message sent Received message buffer length: 59611 Going to top of loop to parse more messagesSending message length: 9222 Resetting buffers Decoded length: 9222 Message sent Sending message length: 5144 Received message buffer length: 9222 Going to top of loop to parse more messages Resetting buffers Decoded length: 5144 Message sent Sending message length: 31213Received message buffer length: 5144 Going to top of loop to parse more messages Resetting buffers
Notice the interleaving of the STDOUT.puts lines. The threads are indeed running together and stepping over each other.
Factoring out the pieces to make things more modular is left as an exercise for the reader.