RabbitMQ

RabbitMQ is a reliable message queue. The RabbitMQ pipeline provides an implementation that imports all BlockchainSpider outputs into RabbitMQ.

To enable this pipeline, you should first install the dependencies:

pip install -r plugins/rabbit/requirements.txt

And add the configuration in BlockchainSpider/settings.py:

ITEM_PIPELINES = {
    'plugins.rabbit.pipelines.RabbitMQPipeline': 666,
}

Next, start your spider command with the specific arguments:

• rabbit_uri: The URI of the RabbitMQ server, e.g., amqp://guest:guest@localhost:5672.
• rabbit_exchange: The RabbitMQ exchange to publish messages to.
• rabbit_routing_prefix: The prefix for the routing key. The route key for each kind of item is prefixed with this prefix.

For example, if you want to import all the data generated by trans.block.evm into RabbitMQ (localhost:5672), you can start the following command:

scrapy crawl trans.block.evm \
-a providers=https://eth.llamarpc.com \
-a start_blk=19000000 \
-a rabbit_uri=amqp://guest:guest@localhost:5672 \
-a rabbit_exchange=BlockchainSpider \
-a rabbit_routing_prefix=trans.block.evm.ethereum

Note that the trans.block.evm Spider outputs SyncItem. If you wish to receive SyncItem via RabbitMQ, then you need to listen to the BlockchainSpider exchange, with the routing key: trans.block.evm.ethereum.SyncItem.