How to Use Onyx for Real-Time Data Processing

Onyx is a distributed, high‑performance data processing platform that handles both stream and batch workloads with a simple functional API. This tutorial walks you through setting up a basic Onyx pipeline, from installation to running your first transformation on a dataset.

Start by installing Onyx via Leiningen or Clojure tools. Add `[onyx “0.14.0”]` to your `project.clj`. Onyx runs on a cluster, but for development you can launch a single‑node environment using `Onyx::Dev` utilities included in the library.

Article illustration

1. Define Your Pipeline’s Workflow

Onyx uses a **catalog** of tasks and a **lifecycle** to orchestrate data flow. Create a catalog vector where each task is a map with `:onyx.core/name`, `:onyx.core/plugin` (e.g., `:onyx.plugin.core-async` for input/output), and `:onyx.core/fn` for your transformation function.

(def catalog
  [{:onyx.core/name :read-messages
    :onyx.core/plugin :onyx.plugin.core-async/input
    :onyx.core/type :input
    :onyx.core/medium :core.async
    :onyx.core/batch-size 10
    :onyx.core/doc "Reads from a core.async channel"}
   {:onyx.core/name :uppercase
    :onyx.core/fn :my-ns/uppercase-fn
    :onyx.core/type :function
    :onyx.core/doc "Transforms text to uppercase"}
   {:onyx.core/name :write-results
    :onyx.core/plugin :onyx.plugin.core-async/output
    :onyx.core/type :output
    :onyx.core/medium :core.async
    :onyx.core/batch-size 10}])

2. Write Your Transformation Function

Each task’s `:fn` must be a namespaced function that accepts a segment (a Clojure map) and returns a segment or collection of segments. For example:

(defn uppercase-fn [segment]
  (update segment :text clojure.string/upper-case))

3. Start the Lifecycle and Submit the Job

Use `onyx.api/start-env` to boot a local instance, then `onyx.api/submit-job` with your catalog and workflow (which defines edges between tasks).

(let [env (onyx.api/start-env {:onyx/id :local-dev})
      job (onyx.api/submit-job
            {:catalog catalog
             :workflow [[:read-messages :uppercase]
                        [:uppercase :write-results]]
             :lifecycle []})]
  ...)

4. Feed Data Through Core.Async Channels

Attach core.async channels to the input and output task names. Push data into the input channel, and pull transformed results from the output channel.

(def input-chan (onyx.plugin.core-async/input-stream job :read-messages))
(def output-chan (onyx.plugin.core-async/output-stream job :write-results))
(put! input-chan {:text "hello"})
(println (take! output-chan)) ;; => {:text "HELLO"}

Onyx automatically distributes work across available cores and persists state via a ZooKeeper‑backed coordinator. Experiment with larger datasets and parallel tasks to leverage the full power of stream‑oriented data processing.

sarah antaboga
Author: sarah antaboga

Leave a Reply

Your email address will not be published. Required fields are marked *