How to Use Onyx for Real-Time Data Processing
Onyx is a distributed, high‑performance data processing platform that handles both stream and batch workloads with a simple functional API. This tutorial walks you through setting up a basic Onyx pipeline, from installation to running your first transformation on a dataset.
Start by installing Onyx via Leiningen or Clojure tools. Add `[onyx “0.14.0”]` to your `project.clj`. Onyx runs on a cluster, but for development you can launch a single‑node environment using `Onyx::Dev` utilities included in the library.
1. Define Your Pipeline’s Workflow
Onyx uses a **catalog** of tasks and a **lifecycle** to orchestrate data flow. Create a catalog vector where each task is a map with `:onyx.core/name`, `:onyx.core/plugin` (e.g., `:onyx.plugin.core-async` for input/output), and `:onyx.core/fn` for your transformation function.
(def catalog
[{:onyx.core/name :read-messages
:onyx.core/plugin :onyx.plugin.core-async/input
:onyx.core/type :input
:onyx.core/medium :core.async
:onyx.core/batch-size 10
:onyx.core/doc "Reads from a core.async channel"}
{:onyx.core/name :uppercase
:onyx.core/fn :my-ns/uppercase-fn
:onyx.core/type :function
:onyx.core/doc "Transforms text to uppercase"}
{:onyx.core/name :write-results
:onyx.core/plugin :onyx.plugin.core-async/output
:onyx.core/type :output
:onyx.core/medium :core.async
:onyx.core/batch-size 10}])
2. Write Your Transformation Function
Each task’s `:fn` must be a namespaced function that accepts a segment (a Clojure map) and returns a segment or collection of segments. For example:
(defn uppercase-fn [segment]
(update segment :text clojure.string/upper-case))
3. Start the Lifecycle and Submit the Job
Use `onyx.api/start-env` to boot a local instance, then `onyx.api/submit-job` with your catalog and workflow (which defines edges between tasks).
(let [env (onyx.api/start-env {:onyx/id :local-dev})
job (onyx.api/submit-job
{:catalog catalog
:workflow [[:read-messages :uppercase]
[:uppercase :write-results]]
:lifecycle []})]
...)
4. Feed Data Through Core.Async Channels
Attach core.async channels to the input and output task names. Push data into the input channel, and pull transformed results from the output channel.
(def input-chan (onyx.plugin.core-async/input-stream job :read-messages))
(def output-chan (onyx.plugin.core-async/output-stream job :write-results))
(put! input-chan {:text "hello"})
(println (take! output-chan)) ;; => {:text "HELLO"}
Onyx automatically distributes work across available cores and persists state via a ZooKeeper‑backed coordinator. Experiment with larger datasets and parallel tasks to leverage the full power of stream‑oriented data processing.