Skip to content

Latest commit

 

History

22 Commits

Folders and files

NameName
Last commit message
Last commit date
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

* CSVX
  A dependency-free tool that enables you to control how to
  tokenize, transform and handle files with char(s) separated values.

  Works in Clojure, ClojureScript (Node.js and browser), and Babashka.

* Usage
Not sure if I am going to publish this to Clojars but if you are using
[[https://clojure.org/guides/getting_started][tools.deps]], you can just add following to deps.edn to add it to your project.
  #+begin_src clojure :results silent
    {:deps
      {github-oneness/csvx
        {:git/url "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/oneness/csvx"
         :sha "LATEST_COMMIT_SHA"}}}
    ;; Then require from your repl like this:
    (require '[csvx.core :as csvx])
    ;; You can just use the one public fn `readx` without any options to parse csv:
    (csvx/readx "resources/100-sales-records.csv")
    ;; Note that csvx/readx takes optional arg where you can pass in following
    ;; options: (listed values here are defaults if no option is given See src/csvx/core.clj
    ;; for details).
    {:encoding "UTF-8"
     :max-lines-to-read Integer/MAX_VALUE  ;; or Number.MAX_SAFE_INTEGER in CLJS
     :line-tokenizer #(.split ^String (str %) ",")
     :line-transformer #(map-indexed hash-map %)}
    #+end_src

**Clojure (CLJ) Usage:**

#+begin_src clojure :results silent
  (require '[csvx.core :as csvx])

  ;; Read a CSV file - returns data directly
  (csvx/readx "data.csv")
  ;; => [[{0 "name"} {1 "age"} {2 "gender"}]
  ;;     [{0 "John"} {1 "32"} {2 "M"}]
  ;;     [{0 "Susan"} {1 "28"} {2 "F"}]]

  ;; Read with options
  (csvx/readx "data.csv"
              {:encoding "UTF-8"
               :max-lines-to-read 100
               :line-tokenizer #(.split ^String (str %) ",")
               :line-transformer #(map-indexed hash-map %)})
#+end_src

**ClojureScript (CLJS) Usage:**

CLJS usage returns a Promise and supports multiple input types:

*** Node.js:**

#+begin_src clojure :results silent
  (require '[csvx.core :as csvx])

  ;; Read from file path - returns Promise
  (-> (csvx/readx "data.csv")
      (.then (fn [data]
               (js/console.log data)))
      (.catch (fn [err]
                (js/console.error err))))
#+end_src

*** Browser:**

#+begin_src clojure :results silent
  ;; From File object (e.g., <input type="file">)
  (-> (csvx/readx file-obj)
      (.then (fn [data]
               (js/console.log data))))

  ;; From URL
  (-> (csvx/readx "https://example.com/data.csv")
      (.then (fn [data]
               (js/console.log data))))
#+end_src

**Babashka Usage:**

Babashka is a native Clojure interpreter for scripting with fast startup.
csvx works with Babashka out of the box with no modifications needed.

Example script (=script.bb=):

#+begin_src clojure :results silent
  #!/usr/bin/env bb

  (require '[csvx.core :as csvx])

  ;; Read a CSV file - returns data directly
  (def data (csvx/readx "data.csv"))

  ;; Process the data
  (prn (count data) "rows read")
  (prn (first data))
#+end_src

To use csvx in a Babashka script, place =src/csvx/core.cljc= on your classpath:

#+begin_src bash
  bb --classpath src script.bb
#+end_src

**Custom Tokenizers and Transformers:**

Following example shows that you can pass in `line-tokenizer` and
`line-transformer` to parse JSON file into Clojure/ClojureScript maps:

#+begin_src clojure :results silent
  (defn decode-json [^String file-path]
    (readx file-path
           {:max-lines-to-read 1
            :line-tokenizer (fn [line]
                              (map #(.split ^String % ":")
                                   (-> (clojure.string/replace line #"\{|\}" "")
                                       (.split ","))))
            :line-transformer (fn [line]
                                (reduce (fn [acc [k v]
                                          (merge acc
                                                 {(-> k read-string keyword) (read-string v)}))
                                        {}
                                        line))}))
#+end_src

* Options

- =:encoding= - File encoding (default: ="UTF-8"=)
- =:max-lines-to-read= - Maximum number of lines to read (default: =Integer/MAX_VALUE= in CLJ, =Number.MAX_SAFE_INTEGER= in CLJS)
- =:line-tokenizer= - Function to split line into fields (default: comma split)
- =:line-transformer= - Function to transform tokenized line (default: map-indexed hash-map)

* Return Values

- **CLJ**: Returns data directly (vector of lines)
- **CLJS**: Returns a Promise that resolves to the data
- **Babashka**: Returns data directly (same as CLJ)

* Develop

#+begin_src bash
git clone https://github.com/oneness/csvx
cd csvx

# Run CLJ tests
clojure -X:test

# Run CLJS Node tests
clojure -X:test-node

# Run CLJS Browser tests (compiles and opens browser)
clojure -X:test-browser

# Run Babashka tests
bb test
#+end_src

* Features

- Works in Clojure, ClojureScript (Node.js and browser), and Babashka
- No dependencies (plain JS interop for CLJS)
- Custom tokenizers for any delimiter-separated format
- Custom transformers for flexible output formats
- Browser support for File objects and URLs
- Configurable line limits for memory-efficient processing
- Babashka compatible - use in scripts for fast CSV processing
- Comprehensive test coverage across all platforms (CLJ, Node, Browser, Babashka)

* Performance: CLJ vs Babashka

Babashka is significantly faster for scripting and CLI usage due to its native binary and instant startup:

| Platform | Cold Start | Warm Start |
|----------|------------|------------|
| CLJ (JVM) | ~3.9s | ~1.0s |
| Babashka | ~0.14s | ~0.025s |

- **Cold start:** Babashka is ~27x faster
- **Warm start:** Babashka is ~40x faster

**Important caveat:** These benchmarks test a small workload (2 tests, simple CSV parsing).
Results should be taken with a grain of salt:

- JVM Clojure's JIT compiler can outperform Babashka for CPU-intensive, long-running tasks
- Babashka's advantage is primarily startup time, not execution speed
- Actual performance depends on workload size, I/O vs CPU operations, and use case
- For large-scale data processing or long-running applications, JVM Clojure may be more suitable

About

A zero dependencies tool that enables you to control how to tokenize, transform and handle files with char(s) separated values in Clojure, ClojureScript and Babashka.

Resources

Stars

23 stars

Watchers

1 watching

Forks

Releases

Packages

Contributors

Languages