{"id":1332,"date":"2026-07-10T18:01:28","date_gmt":"2026-07-10T11:01:28","guid":{"rendered":"https:\/\/sumberlaba.com\/index.php\/2026\/07\/10\/how-to-use-onyx-for-real-time-data-processing\/"},"modified":"2026-07-10T18:01:28","modified_gmt":"2026-07-10T11:01:28","slug":"how-to-use-onyx-for-real-time-data-processing","status":"publish","type":"post","link":"https:\/\/sumberlaba.com\/index.php\/2026\/07\/10\/how-to-use-onyx-for-real-time-data-processing\/","title":{"rendered":"How to Use Onyx for Real-Time Data Processing"},"content":{"rendered":"<h1>How to Use Onyx for Real-Time Data Processing<\/h1>\n<p>Onyx is a distributed, high\u2011performance data processing platform that handles both stream and batch workloads with a simple functional API. This tutorial walks you through setting up a basic Onyx pipeline, from installation to running your first transformation on a dataset.<\/p>\n<p>Start by installing Onyx via Leiningen or Clojure tools. Add `[onyx &#8220;0.14.0&#8221;]` to your `project.clj`. Onyx runs on a cluster, but for development you can launch a single\u2011node environment using `Onyx::Dev` utilities included in the library.<\/p>\n<p><img decoding=\"async\" src=\"https:\/\/via.placeholder.com\/800x600\/4a90d9\/ffffff?text=how%20to%20use%20Onyx%20for%20data%20processing\" alt=\"Article illustration\" style=\"display:block;margin:20px auto;max-width:100%;height:auto;border-radius:8px;\" \/><\/p>\n<h2>1. Define Your Pipeline\u2019s Workflow<\/h2>\n<p>Onyx uses a **catalog** of tasks and a **lifecycle** to orchestrate data flow. Create a catalog vector where each task is a map with `:onyx.core\/name`, `:onyx.core\/plugin` (e.g., `:onyx.plugin.core-async` for input\/output), and `:onyx.core\/fn` for your transformation function.<\/p>\n<pre><code>(def catalog\n  [{:onyx.core\/name :read-messages\n    :onyx.core\/plugin :onyx.plugin.core-async\/input\n    :onyx.core\/type :input\n    :onyx.core\/medium :core.async\n    :onyx.core\/batch-size 10\n    :onyx.core\/doc \"Reads from a core.async channel\"}\n   {:onyx.core\/name :uppercase\n    :onyx.core\/fn :my-ns\/uppercase-fn\n    :onyx.core\/type :function\n    :onyx.core\/doc \"Transforms text to uppercase\"}\n   {:onyx.core\/name :write-results\n    :onyx.core\/plugin :onyx.plugin.core-async\/output\n    :onyx.core\/type :output\n    :onyx.core\/medium :core.async\n    :onyx.core\/batch-size 10}])\n<\/code><\/pre>\n<h2>2. Write Your Transformation Function<\/h2>\n<p>Each task\u2019s `:fn` must be a namespaced function that accepts a segment (a Clojure map) and returns a segment or collection of segments. For example:<\/p>\n<pre><code>(defn uppercase-fn [segment]\n  (update segment :text clojure.string\/upper-case))\n<\/code><\/pre>\n<h2>3. Start the Lifecycle and Submit the Job<\/h2>\n<p>Use `onyx.api\/start-env` to boot a local instance, then `onyx.api\/submit-job` with your catalog and workflow (which defines edges between tasks).<\/p>\n<pre><code>(let [env (onyx.api\/start-env {:onyx\/id :local-dev})\n      job (onyx.api\/submit-job\n            {:catalog catalog\n             :workflow [[:read-messages :uppercase]\n                        [:uppercase :write-results]]\n             :lifecycle []})]\n  ...)\n<\/code><\/pre>\n<h2>4. Feed Data Through Core.Async Channels<\/h2>\n<p>Attach core.async channels to the input and output task names. Push data into the input channel, and pull transformed results from the output channel.<\/p>\n<pre><code>(def input-chan (onyx.plugin.core-async\/input-stream job :read-messages))\n(def output-chan (onyx.plugin.core-async\/output-stream job :write-results))\n(put! input-chan {:text \"hello\"})\n(println (take! output-chan)) ;; => {:text \"HELLO\"}\n<\/code><\/pre>\n<p>Onyx automatically distributes work across available cores and persists state via a ZooKeeper\u2011backed coordinator. Experiment with larger datasets and parallel tasks to leverage the full power of stream\u2011oriented data processing.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>How to Use Onyx for Real-Time Data Processing Onyx is a distributed, high\u2011performance data processing platform that handles both stream and batch workloads with a simple functional API. This tutorial walks you through setting up a basic Onyx pipeline, from installation to running your first transformation on a dataset. Start by installing Onyx via Leiningen &hellip; <\/p>\n","protected":false},"author":2716,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"om_disable_all_campaigns":false,"_monsterinsights_skip_tracking":false,"footnotes":""},"categories":[],"tags":[],"class_list":["post-1332","post","type-post","status-publish","format-standard","hentry"],"aioseo_notices":[],"_links":{"self":[{"href":"https:\/\/sumberlaba.com\/index.php\/wp-json\/wp\/v2\/posts\/1332","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/sumberlaba.com\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/sumberlaba.com\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/sumberlaba.com\/index.php\/wp-json\/wp\/v2\/users\/2716"}],"replies":[{"embeddable":true,"href":"https:\/\/sumberlaba.com\/index.php\/wp-json\/wp\/v2\/comments?post=1332"}],"version-history":[{"count":0,"href":"https:\/\/sumberlaba.com\/index.php\/wp-json\/wp\/v2\/posts\/1332\/revisions"}],"wp:attachment":[{"href":"https:\/\/sumberlaba.com\/index.php\/wp-json\/wp\/v2\/media?parent=1332"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/sumberlaba.com\/index.php\/wp-json\/wp\/v2\/categories?post=1332"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/sumberlaba.com\/index.php\/wp-json\/wp\/v2\/tags?post=1332"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}