Episode 016: When 8 - 1 = 6

Christoph discovers that time creates its own alternate universe.

  • Continuing our exploration of “literate” time logs
  • We want a function to turn the timestamps into minutes.
  • Format: Fri Feb 08 2019 11:30-13:45
  • Keep it simple: extract times with a regex and use math
    • minutes in day = hours * 60 + minutes
    • total minutes = end minutes - starting minutes
  • Problem: What happens when we work past midnight? Negative minutes!
  • We decided to have only one date, so a time before the starting time must span midnight.
  • Format only allows for an activity to be <24 hours.
  • “If we end up doing any activity in our life longer than 24 hours, I think we should that we might have other problems.”
  • Easy Solution: If the end time is before start time, add 24 hours.
  • “When I get past any sort of simpler math, I just type it into my connected editor REPL because I know Clojure can do it faster than I can.”
  • Now we have a function to get minutes, want to add them all up.
  • Use loop and recur to iterate through the array and track the sum.
  • Oh wait, what about Daylight Savings Time?
  • “We all pretend that time is actually in the future.”
  • If it involves dates and times, we can’t just do simple math.
  • “If I do this the right way, I now have to open a whole new can of worms.”
  • Easy way out: write “doesn’t support DST” in the release notes and call it “user error”!
  • “Any time you have to be careful about something, you’re probably doing it wrong.”
  • Use a time API.
  • “The Clojure library is just a wrapper because nobody wants to reinvent this thing.”
  • Java time is immutable, so that works nicely with functional languages. No cloneing!
  • Lots of functions in the time API. Which ones do we need?
  • Our workflow: try out different expressions in a comment block in our connected editor to figure out the specific time functions we need.
  • “Local” dates and times don’t have time zones, but “zoned” dates and times do have them.
  • Need to create timestamps for accurate math: date + time + zone
  • In practice, when we use dates and times, they are implicitly connected to a time zone.
  • Your time zone is your alternate universe: it affects the meaning of your dates and times.
  • We added support for DST without changing the function signature.
  • But, how do we add up other totals? Looks like we’re going to need to change even more.

Episode 015: Finding the Time

Nate spends some time figuring out how to track his time.

  • New problem: Want to track time using a text file.
  • Text files are great:
    • Keyboard oriented: in terminal, in editor
    • Something you can put under revision control
  • Need: date and time range
  • Format of the time stamp should be convenient for us to read and edit.
  • Let the computer do the work to convert the convenient notation to something usable!
  • Format: Fri Feb 08 2019 11:30-13:45
  • One timestamp per line. Oh wait, we need a description. Six minutes in and we’re already changing the requirements!
  • What are “literate” text files? “You can mix human words in amongst data the computer will use to make it more understandable for the humans.”
  • How to find the times? Attempt to match the time format within each line.
  • “It’s kind of like an inverse comment. Instead of every line being valid and you have to comment out lines you don’t want, if it’s in a known format, those are the lines we want and everything else is a comment.”
  • Can use line-seq with clojure.java.io/reader. That uses lazy I/O.
  • Potential Issue: lazy I/O can defer I/O errors to when you’re in the middle of processing the data.
  • Lazy I/O is less of an issue with files and more of an issue with network sockets.
  • Another approach: slurp in all the data at once and call split-lines
  • “Trade a little memory for some safety.”
  • Clojure uses the seq abstraction whichever way you choose. It’s Clojure’s unifying abstraction.
  • “Even maps get squeezed into a seq if you push them hard enough.”
  • Next step: figure out which lines are time entry lines and which lines are commentary
  • Use a regex to match against each string using Clojure’s built-in #"..." form.
  • In other situations, you can use instaparse for grammar-based parsing.
  • Put timestamps on their own line
    • Easier to work with
    • Allows for more error detection (in the future)
  • Use capture groups and re-matches detect the match and extract the parts.
  • Use when-let to destructure and do something only if it matches
  • Better yet, make a function time-info that does the match and returns either nil or the string that matched.
  • For now, just start by using doseq to just go through all the lines and test our matching.
  • Making sense of the timestamp string is a whole new problem for next time.

Episode 014: Fiddle with the REPL

This week Christoph gets some work done by fiddling around.

  • Editor integration is a massive change in how to use the REPL.
  • Put a comment block right underneath a function you are working on, and invoke the function with test arguments.
  • Can quickly make changes to code and try it out–all in the same place!
  • Problem: comment blocks littering the code.
  • “A good rule of thumb I’ve found is: when I have to use the world ‘careful’, it’s a big red flag. I’m doing something I should be able to prevent without having to be careful.”
  • Code in comment blocks can be helpful in the future.
  • ”‘It’s not done yet.” What file is ever done? You’re always going to come back to it in 6 months.”
  • Easy to skip around between forms you’ve run before.
  • “It’s like a random access REPL history.”
  • Ah ha moment: make a separate file for the helpful comment blocks.
  • “[The comment blocks] are like the copilot of my development experience.” “They’re me helping me!” “You’re your own copilot!”
  • Separate file needed a namespace: the fiddle namespace was born.
  • “I’m just fiddling around.”
  • Put the “fiddle files” in the dev tree so it only gets loading in development.
  • “These fiddle files started multiplying, like little helpful rabbits.”
  • Like pages in a notebook centered on a theme. Each “page” is a file:
    • a file for remembering how to call certain core functions
    • a file for experimenting with date and time conversions
    • a file for exploring a new concept
    • a file for interacting with new code being developed
  • Never have to leave to editor. REPL is working behind the scenes.
  • Unlike REPL history, you can share fiddle files.
  • Fiddles can help show the developer’s mindset during development.
  • “It’s like seeing into the back of your mind of the way you explore this on your own.”
  • Example: exploring a data set
    • work on code for parsing a data file
    • write different expressions to sift through the data
    • e.g. threaded expression to: open the data, parse it, filter, and map
    • build up helpers for making sense of the data
    • parser and helpers in fiddle are the start of the “real” application code
    • write examples with commentary
  • Example: exploring the working state of the application
    • set of utility functions for grabbing information from component
    • that set of helpers can be share between developers
    • code that pulls out state from specific components all at once
    • “Where did this nil come from?”
  • Example: exploring external APIs
    • write simple function for calling the HTTP endpoint
    • start calling it with different examples
    • can easily see and re-run calls that you’ve tried
    • Slow API? Use (def response (...)) to save a result, and write expressions using response.
  • “A fiddle file like a scratch pad for new features.”
  • Characteristics of fiddles
    • a working set of expressions
    • like a notebook with code around a theme
    • like a scratch pad for exploring toward a solution
    • can be shared
    • written context from solving the problem
  • “The me of the future won’t have as much access to the mental state of the me of the present.”
  • Do your future self a favor and write down some of your current mental context.
  • Could make educational fiddles for new developers in a project.
  • “Fiddles are a way of capturing process: a way of capturing the craft of making it, not just the outcome.”

Episode 013: Connect the REPL

Nate continues to explore the REPL by gluing it to his editor to see what happens.

  • We revisit Tic-Tac-Toe and change it using our new REPL superpowers.
  • Let’s add request logging to the play endpoint: /play?row=0&col=1.
  • Need to add log/info calls in handler function for play endpoint: on-play.
  • Naive Christoph would add logging to the function, recompile, and run the jar.
  • Less naive Christoph would reload the namespace but still restart the web server.
  • Crazy idea: paste the function definition into the REPL!
    1. Copy the whole defn for on-play
    2. Go to the REPL window
    3. Switch to the app.main namespace (where we put on-play)
    4. Paste the function definition into the REPL
  • No restarting required! Each time we evalute the defn in the REPL, the new function replaces the old version immediately!
  • We can iterate on a single function quickly.
  • Clojure is Lisp, and Lisp is dynamic and isomorphic
    • Dynamic: can change definition on the fly, so the new function exists immediately after executing the definition
    • Isomorphic: everything is a “form”. No special “declaration syntax” vs “expression syntax”. A declaration is a form just like any other expression.
  • “Clojure is dynamic to the end. Every statement is modifying the running application, and because every statement has access to the entire application, all the way down to the definitions, you can redefine on the fly.”
  • Declarations are just ordinary macros (def, defn, defmacro, etc), not special syntax only the compiler understands.
  • A declaration macro updates the enclosing namespace dynamically on the fly!
  • “You just replaced the wheels on the car while you were driving.”
  • “I can replace the steering wheel and my hands don’t even realize they’re using a new steering wheel.”
  • “Running the form is what brings that thing dynamically into existence or replaces the one that was there.”
  • “It’s the way you can speed your development up because you can replace these functions on the fly.”
  • Problem: you have to copy and paste between the editor and the REPL.
  • Idea: use terminal automation to help with the copy + paste!
  • “Yet another example of solving a problem you probably shouldn’t even have.”
  • Pre-Lisp thinking: “The right way to do software is to compile it from grains of sand each and every time, and then run it from the beginning.”
  • Better Idea: Connect the REPL directly to your editor. (See “More reading” for how.)
  • How it works:
    • Move your cursor to the form to evaluate
    • Do the “right” key combo (depends on your editor)
    • Editor sends the form to the REPL
    • REPL evaluates the form and sends the result back
    • Editor shows the result
  • Use comment blocks as a nifty way to provide executable examples within the code.
  • Your editor is a first-class REPL.

Episode 012: Embrace the REPL

Christoph complicated development by misunderstanding the REPL.

  • We go back to the roots of Christoph’s experience with Clojure…
  • How do I run Clojure code?
  • The REPL is fun for evaluation, but how do you run a “real” program?
  • Experience in other languages: edit the file, compile, rerun the program
  • Mentality: get through the edit, compile, run loop as fast as possible!
  • Autobuilder is the logical end.
  • “Where’s my autobuilder in Clojure?!”
  • The REPL model is fundamentally different.
  • “[The REPL] is like cutting the Gordian Knot. You change the problem and that’s how you solve it.”
  • “The reason why the problem I wanted solved, wasn’t ‘solved’, is because nobody has that problem because they do things differently here.”
  • Tactic 1: edit, save, restart the REPL
  • “Restarting the REPL isn’t super slow, but let’s just say it’s not instantaneous.”
  • Tactic 2: edit, save, run (use 'the.namespace :reload) in the REPL
  • “Now I have a REPL history of use statements!”
  • Problems:
    • forgetting to reload dependent namespaces
    • loading dependencies in the wrong order
    • old definitions built up
  • Big Problem: function renames left the old version around, so accidental calls using the old name produced no errors and ran old behavior!
  • Back to quitting the REPL to clean out the cruft. Ugh!
  • Discovered clojure.tools.namespace! Reloads everything and cleans out the cruft!
  • Tactic 3: edit, save, (use '[clojure.tools.namespace.repl :only (refresh)]), (refresh)
  • Problem: refresh would purge itself!
  • “I don’t know why it took me so long to discover the magical user namespace!”
  • The REPL will automatically use the user namespace.
  • “I can put code into a user namespace and it will be there for me.”
  • Christoph would switch namespaces in the REPL without even stopping to wonder what namespace he started in.
  • “It’s like walking out of a door and not even thinking about the fact you’re in a building. Oh wait! What room did I just leave?”
  • Tactic 4: make sure refresh is in the user namespace, edit, save, (refresh)
  • However, Christoph still didn’t understand the REPL.
  • Just thought the REPL was for:
    • reloading the code and restarting
    • evaluating snippets of code
    • inspecting stuff
  • Nate’s big ah-ha moment: “Not only is my application inspectable, it’s fungible. I can change it in place as it’s flying along! I don’t have to restart it!”
  • Christoph’s hint that there was still more came from seeing comment blocks in code and reading about vim-fireplace.
  • “There’s a new room you can explore. That room is editor integration with the REPL.”

Episode 011: The Convention of Configuration

Nate is worried about the hardcoded credentials in the code.

  • It’s episode 011 on 01/11. It’s binary!
  • “As a developer, you quickly learn what’s under your control and what’s not. Very little is under your control.”
  • Don’t accidentally check in the credentials.
  • “We need a configuration management system. Oh wait! That’s a totally different problem.”
  • What about putting the configuration into an EDN file?
  • Let’s call it config.edn
  • Why EDN? Isn’t JSON the “one, true format for config information”?
  • Pros of EDN:
    • native to Clojure
    • can have comments
    • is extensible
  • Why not environment variables?
  • Environment variables are great for production, but in development files are better.
  • Have to restart the REPL to change env variables.
  • Make a component that reads the config. That will reload config when you reset with component.repl.
  • Two main considerations:
    1. Dynamically reloading configuration during development
    2. Plumbing the configuration values through the app
  • Make a namespace for app.config
  • “Files are way more fungible than the environment.”
  • We want both options: env for production and a file for dev.
  • Make two functions in app.config: from-env and from-file.
  • Use schema to make sure both functions return the same config map.
  • “A thing I have done before…you decide if it’s clever.”
  • Define a default configuration and then merge the config maps with that.
  • Better yet, from-env handles defaults and we merge the map from dev.edn into that.
  • Can use environ with Leiningen profiles. Still requires restarting the REPL.
  • Defaults should make sense for development, so you can just check out and run.
  • For bad config, we want helpful error messages, not stack traces.
  • What goes in configuration?
    • Items under your control
    • Items that vary
  • Twitter API URL does not vary.
  • Even if Twitter provided sandbox URLs, those URLs don’t vary. The config option should be “production” and “sandbox”, not the URL. The wrapper code will select the right URL.
  • Avoid the nonsense of trying to infer “sandbox” from reading the URL.
  • “The hallmark of good design is: the less thinking you have to do, the better.”
  • “The more semantic the config, the less thinking you have to do.”
  • It’s important to think about the data first, where it comes from, and where it goes.

Episode 010: From Mud to Bricks

Christoph can’t stand the spaghetti mess in main. Time to refactor.

  • Main function does too much. We want cohesive pieces that each have one job.
  • Two part problem:
    • Too much in the main loop.
    • Main starts and then never stops. Have to exit the repl to restart.
  • We need 1) separate parts that can be 2) started and stopped.
  • Main should just focus on the code needed for command line invocation.
  • Let’s get the parts component-ized!
  • “It’s one thing that I’ve become more sure of in my career is that no application is actually done.” “Useful apps only get more complex.”
  • Internal dependencies are different than external dependencies (“libraries”).
  • Many internal dependencies create high coupling throughout the code.
  • “Once everything starts touching everything you have to understand everything to understand anything.”
  • Like functions use parameters to limit scope, a component is the next level up and uses resource dependences to limit coupling.
  • Each component implements a clear behavior (interface) and can be a resource to other components.
  • Can understand component’s behavior via its interface (and docs) without reading all the code–just like understanding a function through it’s signature and docs.
  • We use the “Component” library to make components in Clojure–has REPL integration too.
  • Components to make:
    1. web server
    2. polling and fetching loop
    3. the core.async channel used between them
  • The Lifecycle interface allows a component to be started and stopped.
    • start must return a reference to the “started” state
    • stop must return a reference to the “stopped” state
    • Gotcha: don’t return a nil!
  • To use Lifecycle, you’ll need to make your component a “record”.
  • Two main goals of Component:
    1. allow stateful components
    2. define dependencies between components
  • Surprise! Any reference can be a “component” as a non-lifecycle dependency.
  • Write a function new-system which returns the component “system map”
  • Mind your names. Make the system map key match a component’s dependent field.
  • “Component is a convenient way of being able to specify all those dependencies in a concise map, so you know this is the intersection of all of my application together.”
  • A component should be able to be understood alone.
  • “Component is like giving you application parts as function parameters. Just like when making a function, you don’t worry about how something gets passed in as a parameter.”
  • You still need to understand each of the parts, but you don’t have to worry about where the part came from.
  • At the top level, you can see all the parts together and how they are connected.
  • Immutability gives you bulkheads between each of your components so you can safely reason about them separately.
  • Use component.repl to start and stop the whole system without restarting the REPL!
  • Need some tooling to keep the main thread from exit. Can use promise, deref and a shutdown handler (see below).
  • “We can keep each ball of mud it its own little basket so all the mud doesn’t ooze together.”

Episode 009: Channeling Re-search

Nate can’t decide what to watch on Twitter, and the app restarts are killing him.

  • The Twitterthon continues. “It’s an infinite stream of Twitter.”
  • Nate wants to watch #clojurescript too.
  • Just change the search string to "#clojure OR #clojurescript"?
  • Should avoid hardcoding the value so we don’t have to recompile every time.
  • Command line argument still requires a restart.
  • Let’s use a curl UI (like Ep 004)!
  • Wait, what?! Why curl?
    • Can send something into the running process
    • Can separate out the client
    • Builds on top of HTTP, so there are lots of tools for testing (like curl itself)
  • Use a URL query string: http://localhost:8000/search?The+search+string
  • “It’s a true query string in the truest sense of the term ‘query.’”
  • “It is actually using the thing for the thing it was meant to be.”
  • How do we get the query string from the webserver thread to the polling loop thread?
  • “This is a perfect case for a global.” Oh wait, that’s mutation.
  • How should we structure the app? The main function must:
    • start the thread for the web server
    • start the thread for the polling loop
  • Specific problem: the /search route handler function needs a way to send the new query string to the polling loop.
  • With mutation: share an atom and give the handler and the polling loop a reference to it.
  • No mutation? Use a core.async channel.
  • A channel allows a thread to pass data to another thread.
  • With no channel buffer, it will synchronize two threads (“rendezvous”).
  • (We lie. There is no peek function.)
  • Problem: polling thread is stuck waiting on the channel, so it stops polling.
  • Solution: Use alt!! to simultaneously listen to the “new search” channel and a timeout channel.
  • What is a timeout channel? A channel that closes after n msecs have passed.
  • New problem: the cache (for de-duplicating) has stale content after changing the query.
  • Solution: same logic that adopts the new search term will reset the cache. (See Ep 007.)
  • Polling loop structure:
    1. fetch
    2. process results
    3. block while listening to the channels
    4. if new search string, recur with new cache and new search string
    5. otherwise, recur with same cache and search string
  • Only want the fetch in one part of the loop.
  • Don’t even need curl. Just type in the URL for the backend on your phone.
  • core.async lets threads coordinate without having to know about each other!

Episode 008: Twitter, Plated

Christoph tries to figure out why Twitter stopped talking about Clojure.

  • “Are you twitterpated?”
  • Building on where we left off last episode.
  • Runs and just stops working.
  • “I was pretty sure it stopped working because people on Twitter just stopped talking about Clojure. After about a day of that, I realized people were talking about Clojure, and I just wasn’t seeing it.”
  • The auth token expired! What should we do?
  • Why should the main loop have to deal with getting a new auth token?
  • “The Twitter wrapper should be concerned with all of the warts and complexities of dealing with Twitter.”
  • “What problems should bubble up, and which ones shouldn’t?”
  • The wrapper should handle the retry.
  • It’s like a kitchen in a restaurant. What are the steps of fulfilling an order? The customer doesn’t care.
  • “There’s a side-effect: the freezer mutates.”
  • The wrapper gets to worry about all the steps:
    • turning the order into the specific request for the kitchen
    • do the I/O to fetch and fulfill the request
    • the “input transform” takes the mass of data and picks out the relevant parts
    • the “internal” version is returned
  • “Like all good metaphors, they stretch to the point where they break, like a rubber band.”
  • Maybe avoid expired tokens by authenticating every time? Too much overhead.
  • If the handle is mutable, then retry can just update the handle with the new token.
  • A mutable handle does allow the wrapper to control the concern.
  • The “handle” is the state of the wrapper. The term “handle” comes from I/O libraries.
  • Instead of mutation, have the search function return [updated-handle, result].
  • search can catch an auth exception, retry, and return a new auth handle.
  • Instead of search retrying, the fetcher can do it! Then it works for all kinds of requests.
  • Better yet, leave fetch simple, and have a fetch-with-retry function that uses fetch.
  • Can have even more policy functions like, fetch-with-retry-forever.
  • “Keep calm, and assoc on.”
  • “I’m never going to miss another Clojure tweet. I’m going to read them all!”

Episode 007: Input Overflow

Nate just wants to print the tweets, but the input wants to take over.

  • Got tweet fetching working last time. Now we want to print out the tweets.
  • API returns a lot information
  • “We should probably not use pprint as our output mechanism.”
  • Using the uniqueness filtering from last time
  • The goal is NOT to write a generic Twitter wrapper.
  • Goal is to write a wrapper that just gives us what our application needs.
  • “I don’t want to spread the complexity of Twitter all over my app. I want to hide it inside of the wrapper.”
  • Clojure data structures are generic enough already!
  • We pick out just what we need and put it in our own data structure.
  • We use our “internal” structure throughout the application code.
  • Our internal structure protects our application code from changes in Twitter’s format.
  • Keep the structure minimal and grow it when the application code needs more fields.
  • Where should we put the internal model? Make it a part of the wrapper.
  • A wrapper should expose a data model
  • A “sequencing” function just threads steps defined by other functions. Pure steps are easy to test, so make the I/O steps as minimal as possible.
  • Technique is called “pushing I/O to the edges”.
  • Let any exceptions fly back to the main loop. It can sleep a little and retry.
  • Separate out formatting from println, so you can unit test output.
  • Put the cache behind its own data model
  • The cache data model provides logical operations like (defn filter-new [cache tweets] ...)
  • The filter-new function would return [cache new-tweets]. Cache could have changed.
  • How do you know what’s in the data model exposed by the wrapper?
  • “What is in this thing that we’re passing around? You don’t just want to read the code and figure it out?”
  • “You rapidly get to the point where you don’t know what’s in your data structure.”
  • Use schema.
  • Put the schemas the wrapper exposes in its own namespace.