First paragraph.

My post with code

As more and more teams and companies adopt Apache Kafka, you can find yourself wanting to share data via replication of one or more topics from one cluster to another. While replication of an entire cluster with all of it’s topics as a means of failover can be achieved with tools such as Mirror Maker and Confluent Replicator, for replication of a single topic there are fewer examples. Even more so when the source topic is serialized with Avro, with the schema stored in Confluent Schema Registry.

Here we present a minimal consumer that replicates a single Avro serialized Kafka topic from one cluster to another, while ensuring (only) the necessary Avro schema is registered in the target cluster Schema Registry.

In Clojure we can take or drop elements from a collection based on a predicate using the functions take-while and drop-while. With the function take-while we take elements as long as the predicate returns true. Once the predicate returns false the function stops returning elements. Using the function drop-while we skip elements in the collection if the predicate returns true. If the predicate returns false the remaining elements in the collection are returned.

In the following example we use take-while and drop-while with different collection types:

The map data structure is used a lot in Clojure. When we want to use Java objects in our Clojure code we can convert the Java object to a map with the bean function. This function will use reflection to get all the properties of the Java object and converts each property with the property value to a key with value in the resulting map. The bean function will not recursively convert nested objects to a map, only the top-level properties are turned into key value pairs.

We see several examples of using the bean function in the following code snippet:

It is very easy to work with Java classes in Clojure. If we want to create a new object based on a Java class and invoke methods to initialize the object directly we can use the doto macro. The first argument is an expression to create a new object and the rest of the arguments are functions to invoke methods on the newly created object. The object returned from the first argument is passed as first argument to the method invocations. The doto function returns the object that is created with the first argument.

In the following example code we use the doto function in several cases:

The clojure.string namespace contains a lot of useful functions to work with string values. The escape function can be used to replace characters in a string with another character. The function accepts as first argument the string value and the second argument is a map. The map has characters as key that need to be replaced followed by the value it is replaced with. For example the map {\a 1 \b 2} replaces the character a with 1 and the character b with 2.

In the following example code we use the escape function in several cases:

When we use a function as argument for the map function that returns a collection we would get nested collections. If we want to turn the result into a single collection we can concatenate the elements from the collections by applying the concat function, but we can do this directly with the function mapcat. The function mapcat takes as first argument a function (that returns a collection) and one or more collections as next arguments.

In the following examples we see several uses of mapcat:

Since beginning of time mankind has been looking for a way to separate right from wrong. Where the primeval man judged righteousness by the contributions of the tribe, the current day programmer judges right by the wishes of the customer. For many years the average programmer wrote a bunch of logic to check if the boundaries defined by the client where uphold. As time went on and programming languages involved, metadata could be added to enrich functions, methods, classes and the like.

Of course for Java, these metadata are called annotations. Very soon they were used for a lot of things. Surpressing warnings, managing transactions, building XML/JSON structures and injecting dependencies. And, as you might have guessed by now, validating objects by a set of specific rules. One of the most commonly used frameworks would be the Jakarta Bean Validation framework. But what if I told you the provided annotations of that framework could be very easily expanded.

When we are working with sets in Clojure we can use some useful functions from the clojure.set namespace. In a previous post we learned how we can get the difference of several sets. To get the union of several input sets we use the union function of the clojure.set namespace. The function returns a new set that is the union of unique elements from the input sets. A nil value is ignored by the union function.

In the following example code we use union: