Smart API integrations with Python and Zato

As the number of applications and APIs connected in a cloud-driven world rises dramatically, it becomes a challenge to integrate them in an elegant way that will scale in terms of the clarity of architecture, run-time performance, complexity of processes the systems take part in, and the level of maintenance required to keep integrated environments operational.

Organizations whose applications grow organically with time tend to get entangled in a cobweb of unmanageable dependencies, unidentified requirements, and hidden flows of information that cannot be touched lest seemingly unrelated parts suddenly stop functioning. This can happen to everyone, and is actually to be expected to some extent.

It's natural to think that one can easily manage just a couple of APIs here and there.

Yet, what starts out as just a few calls to one system or another has an intriguing characteristic of inevitably turning into a closely coupled network of actors whose further usage or development becomes next to impossible:

This has become particularly evident in the today's always-connected landscape of online APIs that grow in importance on a huge scale.

Introducing IRA services

To deal with demand and introduce order one can look to software such as the Python-based Zato integration platform, released under LGPL and freely available both from GitHub as well as a set of OS-specific packages.

Zato promotes clear separation of systems and APIs being integrated and emphasizes architecting integrations out of IRA services that substitute point-to-point communication.

An IRA service is a piece of functionality running in a distributed and clustered environment with the attributes of being:

• Interesting
• Reusable
• Atomic

This, in fact, is nothing else than the Unix philosophy taken onto a much higher level of composing processes out of applications and APIs rather than individual system programs.

Different settings three decades after the philosophy has been originally postulated yet the principles stay the same—be composable instead of tying everything into a monolith.

While designing software as reusable and atomic building blocks is understood, being interesting may raise an obvious question—what does it mean to be interesting?

The answer is a two-fold question again:

• Would you truly accept to use such a service yourself each and every day for the next 10 years or more?
• Can you fully explain the service's purpose to non-technical stakeholders, the people who ultimately sponsor the development, and have them confirm they can clearly understand what values it brings to the equation?

If the stakeholders happen to be technical people, the second question can be reworded—would you be able to explain the service's goal in a single Tweet and have half of your technically-minded followers retweet or favorite it?

Looking at it through a Unix philosophy's perspective and command line tools, this is interesting:

• Are you OK with using the ls command? Or do you strongly feel it's a spawn of R'lyeh that needs to be replaced as soon as possible?

• Would you have any issues with explaining what a purpose of the mkdir command is to a person who understands what directories are?

And now, what is not interesting:

• Would you be happy if all shell commands had, say, combinations of options expressed in digits only, changed weekly and unique to each host? For instance 'ls -21' instead of 'ls -la' but 'ls -975' for 'ls -latrh'? I know, one could get used to everything, but would you truly condone it with a straight face?

• How would you explain without any shame the very existence of such a version of ls to a newcomer to Linux?

Same goes for integrating APIs and systems—if you follow the IRA principles you'll be asking yourself the same sort of questions. Add reusability and atomicity on top of it and you've got a recipe for a nice approach to connecting otherwise disconnected participants.

Such a service can also be called a microservice.

Implementing IRA services

Now let's suppose there's an application using OpenStack Swift to store information regarding new customers and it all needs to be distributed to various parties. Here's how to approach it while taking IRA into account:

• Have the producer store everything in Swift containers
• Use Zato's notifications to periodically download latest sets of data
• Have Zato distribute the information to intended recipients using given recipient's chosen protocols

All of the IRA postulates are fulfilled here:

• Producer simply produces output and is not concerned with who really consumes it—if there are more recipients with time, nothing really changes because it's Zato that will know of it, not the producer

• Likewise, recipients can conveniently assume the fact they are being invoked means new data is ready. If there's a new producer with time, it's all good, they will just accept the payload from Zato.

• It's Zato that translates information between various formats or protocols such as XML, JSON, SOAP, REST, AMQP or any other

• Hence, the service of notifying of new customers is:

  • Interesting—easy to explain
  • Re-usable—can be plugged into various producers or consumers
  • Atomic—it does one thing only and does it well

from zato.server.service import Service

class CreateCustomer(Service):
    def handle(self):

        # Synchronously call REST recipients as defined in Redis
        for conn_name in self.kvdb.conn.smembers('new.customer'):
            conn = self.outgoing.plain_http[conn_name].conn
            conn.send(self.cid, self.request.raw_request)

        # Async notify all pub/sub recipients
        self.pubsub.publish(self.request.raw_request, '/newcust')

This is yet another example of using IRA in practice because Zato's own architecture lets one develop services that are not bothered with details of where their input comes from—most of the code above can be re-used in different contexts as well, the code itself won't change.

The rest is only a matter of filling out a few forms and clicking OK to propagate the changes throughout the whole Zato cluster.

That code + a few GUI clicks alone suffices for Swift notifications be distributed among all the parties interested though on top of it there is also command line interface and the platform's own public admin API.

And here it is, an IRA service confirming to IRA principles:

• Interesting—strikes as something that will come in handy in multiple situations
• Reusable—can be used in many situations
• Atomic—does its own job and excels at it

Such services can now form higher-level business processes, all of them again interesting, reusable and atomic—the approach is scalable from lowest to highest levels.

To get in touch with the Zato project, you can drop by the mailing list, IRC, Twitter or the LinkedIn group.

Everyone is strongly encouraged to share their thoughts, ideas or code on how to best integrate modern APIs in a way that guarantees flexibility and ease of use.