Ep 120: Data Vision

► Play Episode

Each week, we discuss a different topic about Clojure and functional programming.

If you have a question or topic you'd like us to discuss, tweet @clojuredesign, send an email to feedback@clojuredesign.club, or join the #clojuredesign-podcast channel on the Clojurians Slack.

This week, the topic is: "ingesting, deciding, and merging". We curate data, merge it, and represent choices.

Our discussion includes:

Selected quotes

A pure data model is data that models something in the real world and pure functions you pass that data through. That's it.

For an ingested model, you want to curate a subset of external data, or normalize it in some way, so it's useful to the rest of your application.

Two main pieces: a representation of the information that is pertinent to your application, and a set of functions that operate on that data. Those functions tend to fall into patterns.

The ingested model is a coherent window to that outside world. It's the lens through which your application sees that external data. It serves as a stabilizing force for the rest of your application.

The Kubernetes data is wide, and it's deep. It's got a lot of information. It's only rivaled by GitHub responses in its size and scope!

The decision, itself, is data. If you want to push side effects to the edges, you need to represent the result of a decision in some way.

You put the side effects on the edges. I/O on the way in. I/O on the way out. The pure models in the middle.

The code that connects the model to a side effect is as simple as taking things out of a map and passing them as arguments into a side effect function. It only does one thing. It's almost one line of code. It's so small that it's easy to understand.

Most of your mind is focused on the ingested model and the decision-making model. You're curating and representing information, and then you're reasoning about that information.

A merging model is a pure model that takes other pure models and creates a composite, or enriched, singular output. It's multiple pieces stitched together into a larger work.

Merging models together is a separate concern from the individual models. A merging model's job is to construct a cohesive view from separate, but related information.

If it's all pure, you can test it. You can make sure that it behaves the correct way in different situations.

A way to look at at different models is by responsibilities. The outside system takes responsibility for representing the data in a way that has utility for its concerns, design constraints, and realities. The ingested model's responsibility is figuring out what matters for the application. The decision-making model's responsibility is representing choices that matter and the consequences of those choices. The merging model's responsibility is about how to connect separate things into a larger whole

So each kind of model is a separate problem space, but they're all related, and it becomes clear how they are connected. By decomposing models into separate problem spaces, you end up with models that are relatively stable because they only need to change when that aspect of the problem changes. They are decoupled and orthogonal to each other because they're in different information domains, problem domains, responsibility domains.

One of the best parts of a decoupled system is that it's easy to understand one model at a time. The theme of this series is "re-entrant coding". I only want to understand the parts of the system that I have to understand to accomplish the current task. I don't want to have to understand everything altogether. By decoupling these things with clean responsibilities and boundaries, we get a system that's easy to start working with again.

Each model is visible because it takes data and produces data. You have something that can be seen, inspected, logged, recorded, and put in a test. You avoid the kind of functional composition where information goes through a bunch of internal transformations, and you can't quite get a hold of it in your head. You have a relatively shallow call stack because each model produces data. The orthogonality of the models helps keep the call stack shallow.

A nice, related set of pure models: ingesting, making choices, and merging things together.

You can make the functional jokes. I'll stick with the dysfunctional jokes.