Ep 030: Lazy Does It

Play Episode

Christoph’s eagerness to analyze the big production logs shows him the value of being lazy instead.

Message Queue discussion:

Related episodes:

Clojure in this episode:

Code sample from this episode:

(ns devops.week-02
  (:require
    [clojure.java.io :as io]
    [devops.week-01 :refer [parse-line parse-details]]
    ))


; Parsing and summarizing

(defn parse-log
  [raw-lines]
  (->> raw-lines
       (map parse-line)
       (filter some?)
       (map parse-details)))

(defn code-357-by-user
  [lines]
  (->> lines
       (filter #(= :code-357 (:kind %)))
       (map :code-357/user)
       (frequencies)))


; Failed Attempt: returning from with-open

(defn lines
  [filename]
  (with-open [in (io/reader filename)]
    (line-seq in)))

(defn count-by-user
  [filename]
  (->> (lines filename)
       (parse-log)
       (code-357-by-user)))

; Throws IOException "Stream closed"
#_(count-by-user "sample.log")



; Works, but I/O is coupled with the logic.

(defn count-by-user
  [filename]
  (with-open [in (io/reader filename)]
    (->> (line-seq in)
         (parse-log)
         (doall)
         (code-357-by-user))))

#_(count-by-user "sample.log")



; Separates out I/O. Allows us to compose the processing.

(defn process-log
  [filename f]
  (with-open [in (io/reader filename)]
    (->> (line-seq in)
         (f))))

; Look at the first 10 lines that parsed
#_(process-log "sample.log" #(->> % parse-log (take 10) doall))

; Count up all the "code 357" errors by user
(defn count-by-user
  [filename]
  (process-log filename #(->> % parse-log code-357-by-user)))

#_(count-by-user "sample.log")

Log file sample:

2019-05-14 16:48:55 | process-Poster | INFO  | com.donutgram.poster | transaction failed while updating user joe: code 357
2019-05-14 16:48:56 | process-Poster | INFO  | com.donutgram.poster | transaction failed while updating user sally: code 357
2019-05-14 16:48:57 | process-Poster | INFO  | com.donutgram.poster | transaction failed while updating user joe: code 357