Episode 007: Input Overflow
Nate just wants to print the tweets, but the input wants to take over.
- Got tweet fetching working last time. Now we want to print out the tweets.
- API returns a lot information
- “We should probably not use
pprintas our output mechanism.”
- Using the uniqueness filtering from last time
- The goal is NOT to write a generic Twitter wrapper.
- Goal is to write a wrapper that just gives us what our application needs.
- “I don’t want to spread the complexity of Twitter all over my app. I want to hide it inside of the wrapper.”
- Clojure data structures are generic enough already!
- We pick out just what we need and put it in our own data structure.
- We use our “internal” structure throughout the application code.
- Our internal structure protects our application code from changes in Twitter’s format.
- Keep the structure minimal and grow it when the application code needs more fields.
- Where should we put the internal model? Make it a part of the wrapper.
- A wrapper should expose a data model
- A “sequencing” function just threads steps defined by other functions. Pure steps are easy to test, so make the I/O steps as minimal as possible.
- Technique is called “pushing I/O to the edges”.
- Let any exceptions fly back to the main loop. It can sleep a little and retry.
- Separate out formatting from
println, so you can unit test output.
- Put the cache behind its own data model
- The cache data model provides logical operations like
(defn filter-new [cache tweets] ...)
filter-newfunction would return
[cache new-tweets]. Cache could have changed.
- How do you know what’s in the data model exposed by the wrapper?
- “What is in this thing that we’re passing around? You don’t just want to read the code and figure it out?”
- “You rapidly get to the point where you don’t know what’s in your data structure.”
- Use schema.
- Put the schemas the wrapper exposes in its own namespace.
Parts of a data model
- the data structure: what that model needs to store or represent
- a schema (or spec) to define and validate the data structure
- “constructor” functions that take args and return a data structure
- “update” functions that take the data structure and return a modified one
- “output” functions that take the data structure and return some other representation
Parts of a wrapper
- Application-facing functions. Sole job is to sequence internal wrapper functions.
- Data model for a Twitter API “request”
- Functions that execute a request and return raw data
- Data model exposed by the wrapper
Clojure in this episode:
- lists as “tuples”