Dapr

Nemesis 2.0 makes heavy use of Dapr, the Distributed Application Runtime. The Dapr components that Nemesis utilizes are detailed in the following sections. Images in this page were pulled from the appropriate locations from the Dapr Documentation.

Pubsub

Nemesis utilizes the Dapr Publish & subscribe building block for its internal queueing system. Currently, Nemesis utilizes RabbitMQ for the queue, but this can easily be easily swapped for alternative systems like Kafka or Redis Streams by ensuring the provider is stood up in the docker-compose.yml, modifying the pubsub.yaml file with an alternative provider, and ensuring the connection string is passed through via an environment variable as in the current pubsub.yaml example.

Workflows

Dapr Workflows enable developers to build reliable, long-running business processes as code. They provide a way to orchestrate microservices with built-in state management, error handling, and retry logic for complex distributed applications.

Nemesis uses in two specific places/services. First, in the file_enrichment project, Dapr workflows are used to control the main file enrichment processing logic. The enrichment_workflow() function controls the main enrichment workflow, with the enrichment_module_workflow() function invoked as a child workflow.

The document_conversion project also implements a Dapr workflow in the document_conversion_workflow() function to handle converting documents and extracting text. This is broken out into a separate project as it's a time-consuming task.

Secrets

Nemesis uses the Dapr Secrets management building block to protect secrets internally (like Postgres connection strings). Currently the Local environment variables component is used. These secrets are also refereced within some Dapr files such as pubsub.yaml.

This reason for using this abstraction is so alternative secret management systems like Vault or Kubernetes secrets can be used in the future:

An example of retrieving a secret is at the top of the the housekeeping code to retrieve the POSTGRES_CONNECTION_STRING string.

Service Invocation

In a few places in Nemesis, Dapr's Service Invocation building block is used to ease the complexity of some API invocations. This building block is specifically used when calling the Gotenberg API and when calling some of the internal file enrichment APIs by the web API.