ZeroMQ ---- ZeroMQ Jonathan Rockway http://www.jrock.us/ https://github.com/jrockway/ jon@jrock.us ---- Messaging ---- Roll your own ---- Sockets ---- TCP ---- Stream ---- But we want messages ---- Datagram ---- UDP ---- But we want reliability ---- Protocols ---- Netstrings ---- Incremental JSON ---- SOAP ---- Complexity ---- Bugs ---- Reinventing the wheel ---- But it's a fun wheel to reinvent ---- So that is done. ---- But wait. Peer to peer? ---- Message distribution ---- Publish Subscribe ---- More code ---- More bugs ---- What if there was a library ---- An intelligent transport layer? ---- ZeroMQ ---- ZeroMQ The intelligent transport layer ---- Send messages Not datagrams Not streams ---- Complex topologies ---- No central server ---- Only peers ---- Edge to edge ---- Use cases ---- Publish subscribe ---- Generate data ---- Process elsewhere ---- Start with 1 subscriber ---- Add more for free ---- No code changes ---- No app restarts ---- Durable publisher ---- Subscriber dies ---- Gets message when it comes back ---- No central server ---- Just Works ---- Push Pull ---- Work queue ---- Generate messages ---- Send to available worker ---- Network independent ---- Add new nodes, use TCP ---- Add new processes, use UNIX ---- Add new threads, use inthread ---- Request Reply ---- Acknowledge messages ---- Real-world example ---- Ever use mod_perl ---- Hard to maintain ---- Perl only ---- But fast ---- The web server knows Perl ---- What if anything could be that efficient? ---- Mongrel2 ---- Mongrel2 https://github.com/zedshaw/mongrel2 ---- Web server with handlers ---- As ZeroMQ peers! ---- Push/pull for requests ---- Publish/subscribe for responses ---- Web app slow? ---- Add more app instances ---- Mongrel2 slow? ---- Add more m2 instances ---- Development ---- Server restarts ---- Dead page ---- Hit reload ---- Very annoying ---- Mongrel2 ---- Don't restart ---- Accepts request ---- Waits for app to start ---- Then asks app to handle request ---- Then you get the response ---- My $work_app ---- Monitoring ---- Services collect data ---- Services process data ---- How to connect? ---- ZeroMQ ---- UNIX sockets for production ---- Each service has own process ---- Something segfaults? Auto-restart ---- No data lost ---- Test suite ---- Inproc ---- No need to fork ---- Apps do not care about coordinating ---- ZeroMQ does all that ---- Using it from Perl ---- use ZeroMQ::Raw; ---- Also DMAKI module ---- Will be consolidated soon ---- Simple example ---- use ZeroMQ::Raw; use ZeroMQ::Raw::Constants qw(ZMQ_PUB); my $c = ZeroMQ::Raw::Context->new( threads => 1 ); my $pub = ZeroMQ::Raw::Socket->new( $c, ZMQ_PUB ); $pub->bind('tcp://*:1234'); my $msg = ZeroMQ::Raw::Message->new_from_scalar('hello, world'); $pub->send($msg); ---- use ZeroMQ::Raw; use ZeroMQ::Raw::Constants qw(ZMQ_SUB ZMQ_SUBSCRIBE ZMQ_IDENTITY); my $c = ZeroMQ::Raw::Context->new( threads => 1 ); my $sub = ZeroMQ::Raw::Socket->new( $c, ZMQ_SUB ); $sub->setsockopt(ZMQ_IDENTITY, 'sub1'); $sub->setsockopt(ZMQ_SUBSCRIBE, q{}); $sub->connect('tcp://localhost:1234'); my $msg = ZeroMQ::Raw::Message->new; $sub->recv($msg); print $msg->data; ---- API looks exactly like C docs ---- Except function( object, arg, arg ) is $object->function($arg, $arg) ---- Publisher need not bind ---- Subscriber need not connect ---- $sub->bind('tcp://*:1234'); $pub->connect('tcp://localhost:1234'); ---- Same message flow but allows alternative architecture ---- Bind or connect multiple times ---- $sub->bind('tcp://*:1234'); $sub->connect('tcp://publisher:1234'); ---- Gets messages from connected publishers, and from tcp://publisher:1234 ---- Also possible ---- ipc:// - UNIX sockets inproc:// - internal shared state epgm:// - pretty good multicast (udp) ---- Blocking by default ---- But blocking is less common with ZeroMQ ---- (threads behind the scenes) ---- Can be nonblocking with ZMQ_NOBLOCK ---- Standard POSIX errors (EAGAIN / EWOULDBLOCK) ---- Errors raised as exceptions (and $! set) ---- That's all you need to do to have reliable end-to-end messaging ---- AnyEvent ---- Blocking sucks ---- Being able to run many "things" at a time is nice ---- ZMQ makes this easy ---- AnyEvent::ZeroMQ ---- Integrate ZeroMQ with AnyEvent ---- (ZeroMQ has its own event loop, but we don't want that because it's only for ZeroMQ) ---- Very low-level AnyEvent::ZeroMQ->io ---- $w = AnyEvent->io ... $w = AnyEvent::ZeroMQ->io ... ---- Edge-triggered, not level-triggered. Be careful. ---- use AnyEvent::ZeroMQ::Handle; ---- my $c = ZeroMQ::Raw::Context->new( threads => 1 ); my $s = ZeroMQ::Raw::Socket->new( $c, ... ); ---- my $h = AnyEvent::ZeroMQ::Handle->new( socket => $s, on_read => sub { ... }, identity => 'my socket', ); $h->bind( ... ); $h->connect( ... ); $h->clear_on_read; $h->push_read( sub { ... } ); $h->push_write( 'write this later' ); ---- Uses API roles AnyEvent::ZeroMQ::Handle::Role ::Readable ::Writable ---- Specific types ---- AnyEvent::ZeroMQ::Publish AnyEvent::ZeroMQ::Subscribe AnyEvent::ZeroMQ::Push AnyEvent::ZeroMQ::Pull AnyEvent::ZeroMQ::Request AnyEvent::ZeroMQ::Reply ---- use AnyEvent::ZeroMQ::Publish; my $p = AnyEvent::ZeroMQ::Publish->new( identity => 'mypub', bind => 'tcp://*:1234', connect => 'tcp://more:1234', ); $p->publish( 'a message' ); $p->push_write( 'same thing' ); EV::loop; # or whatever ---- use AnyEvent::ZeroMQ::Subscribe; my $s = AnyEvent::ZeroMQ::Subscribe->new( identity => 'mysub', connect => 'tcp://localhost:1234/', topics => [qw/CATS LOL.CATS/], ); $s->push_read( sub { my $msg = shift; print "got message about cats: ". $msg->data; }); $s->on_read( sub { ... } ); EV::loop; ---- ZeroMQ ---- Messaging made easy ---- Messaging made reliable ---- Get it from CPAN and forget about your transport layer! ----