Integrations Primer: Do, Re, Mi

What is an Integration?

Let’s start at the very beginning. It’s a very good place to start. An “Integration” refers to any mechanism by which two systems share data or issue one another commands. Simple, right?

Where are integrations used?

Building an integration is the right answer when any combination of the following 4 objectives needs to be accomplished:

⦁ Push data from one system into another application or system
⦁ Push data from another application or system into a single system
⦁ Query another system or application for data to pull into one system
⦁ Query one system for data to pull into another application or system

Some integrations are in common use:

⦁ Loading users from Active Directory via LDAP
⦁ Connecting to an SSO (Single Sign-on) provider
⦁ Real-time information sharing of data with another system
⦁ Scheduled imports from data sources (FTP, CSV, Excel, etc…)

It’s becoming increasingly rare to find anyone that isn’t using at least one integration.

Key Terms

There are several terms relevant to integrations and knowing them can help you speak to a others more effectively when you’re getting into integrations. This is a semi-complete glossary of the most important terms:

Design

• JSON – JSON (JAY-Sawn*) is JavaScript Object Notation is a way to display a much easier way to send and receive data for an Integration compared to XML. JSON uses Name-Value pairs which is easier to understand and develop on compared to XML. It has a defined structure of Object, Array, String to Parse, or a combination of all. You can have Objects inside the Array, and one of the values of the Object is a JSON string to parse.
  • Example Object: [ { “id”: 1, “first_name”: “John”, “last_name”: “Smith”, “email”: “johnsmith@example.com” } ]
  • Example String: “[ { \”id\”: 1, \”first_name\”: \”John\”, \”last_name\”: \”Smith\”, \”email\”: \”johnsmith@example.com\” } ]”
  • *It was originally pronounced Jason, but that confuses people with that name. JAY-Sawn is more used.
• XML – Extensible Markup Language uses Tags which are more complex and require additional data to be sent. Similar to HTML, XML uses a Tag based structure. This requires an Open and a Close tag, but it provides the ability to maintain transactions and options inside the tags, which can be beneficial for certain integrations. We would still recommend JSON over XML.
  • Example: <tag></tag>
• Web Services – The Umbrella term for different types of integrations and their Technology being used. For example, Amazon Web Services (AWS) has many services that run off of different ports and services. It includes REST, SOAP/WSDL, XML-RPC/JSON-RPC, and others that aren’t used as much.
• REST – REpresentational State Transfer (or RESTful services) is an Architectural Style that is the most used form of Integration on the Internet today. There are pros and cons, but the pros outweigh the cons. There are millions of REST API calls being made every second, without most people ever knowing. REST uses JSON to process by default, but can use XML. REST is Stateless, which means that it only knows the information I send it. It forgets what was sent previously, so we assign an Unique ID to the request to make sure it knows what to do next. Each Record needs to be sent individually unlike SOAP which can send multiple messages in an envelope at a time. If I were to create a User, before I can request an Update the user Birthday, I need to send the Request with the User ID. The system will not know which record I just created.
• SOAP – Is another type of integration which used to stand for (Simple Object Access Protocol), but it just goes by SOAP now. It seems that once REST came out, the Simple part was hard to justify! Uses XML to process. We won’t cover SOAP in the Beginner Course. Think about SOAP like a Chain Letter you are sending in the mail. we put the letter in an Envelope and addressed it to someone. we then realized we have a few more letters to send, so we put another envelope together. we add those letters to a Manilla Envelope and give it to the Post Office. That is essentially SOAP. You are packaging Messages into an Envelope which needs to be addressed in a very specific way and provide a Stamp to track the letters. Example: Nesting Dolls.
• JDBC – The abbreviation for Java Database Connectivity, which if you ignore the first and last words, Database is the words we need to know. Do you have a MySQL, Oracle, or MSSql Database that you need to pull data from? JDBC is the way to access those tables.

Urls

• Endpoint – The Full Url of the REST API – Protocol + Hostname + Resource (Namespace + Version + Object + Action + ID + Parameters)
  https://harmedia.rest/colors?color=red
  https://harmedia.rest/users/46
• SSL and/or TLS – SSL is the older version of TLS, but is the eponym for TLS. Both describe network-level encryption of data. These are the “S” (secure) in “HTTPS”
• (x509) Certificates – This term refers to a document that stores cryptographic data proving the identity of the holder. These can be used with authentication, encryption, etc. “SSL Certificates” are required for
• Protocol – Most likely you will use: HTTPS (https://) or HTTP (http://). Other protocols you may use: -SSH, -FTP, -SCP. It’s like if you were to pick up the Phone and Dial a Fax number. It may connect, but it doesn’t know how to handle another protocol. Each of them run on a completely different port and require separate programming.
• Port – The Ports are connections that can be made to different openings in the system.
• Hostname – The Main Url of the REST API – Example: api.harmedia.rest or google.com
• Resource – The Relative Url of the REST API. This is the Full Url without the Hostname (Some of the pieces of the Resource may or may not exist for) – Namespace + Version + Object + Action + ID + Parameters
• Namespace – Namespace is basically the Structure you will base your integration on. ‘/api/’ or ‘/catalog’ or ‘wp-json’ are examples.
• Version – Versioning is an important part of the process. If you have a version from the start, you can use the correct version /v1/ and then you can provide different functionality for version 2 /v2/
• Object – The Object is the ‘category’ that you are trying to address. So, if you were trying to take an Action on Users, the Object would be User. If the Category is Products, the Object would be Product
• Action – The Action is the…action you are trying to perform on the Object. If the Object is User and the Action is Create, you will expect all information under that action to be based on the Create functionality. The action could be Search or Edit or Assign. Whatever it is, it should represent what is happening.
• ID – The ID is the specific Record you are trying to address. So, if the User ID is ’34’, the ID will be /34/, but the ID can be letters, numbers, or special characters, such as ‘Eqa2nAAhHN0’
• Query Parameters – Next time you Google something, take a look at the Url you can compare your results. I typed “REST” in Google. Everything after the Question Mark (?) is the Query Parameters. Each Query Parameter is separated by an ampersand (&). For each query parameter, you can loop through the results in the address and use each parameter as a different option.
  Host: google.comAction: search
  Query Parameters: ?q=REST&rlz=1C1RXQR_enUS1038US1038&oq=REST+&gs_lcrp=EgZjaHJvbWUyBggAEEUYOTI…

Content

• Headers – A way to control the specific details of a Request or Response prior to sending the call. Similar to *67
• Body – The information being sent, like your email address or a Message you are sending on the internet.
• Content-Type – Sometimes, you are provided with the wrong format that you need to convert, you can change below.
  Content-Type: application/json
  Content-Type: application/xml
  Content-Type: text/plain
  Content-Type: multipart/form-data; boundary=something
• Accept – What should we return the result as. This becomes helpful if the response comes back as XML but your system only supports JSON. You can send the Content-Type over as XML, and Accept JSON.
  Accept: application/json
  Accept: application/xml
  Accept: text/plain
• Request – The information being sent to the Url/Endpoint to REQUEST an action to be fulfilled. If I were to say “Marco”, what would you say back to me? Polo, right? The Request is the Marco and the Polo is the Response. If I say Ping, your response should be Pong.
• Response – The Response, is the answer back from the other System, letting you know the Result of your Request. Speaking of Ping, on the Internet, a Ping is a request. If I were to say “Marco”, what would you say back to me? Polo, right? The Request is the Marco and the Polo is the Response. If I say Ping, your response should be Pong.
• Status Code – Status codes are your Success and Error Messages, but since there would be way too much text to surround the message, we revert to Numbers which are easier to remember. While most of the websites you visit will be Status Code: 200, you will eventually find a page that is wrong and you will get a 404: Page Not Found.
• CSV – Data Import: Comma Separated Values (CSV) have been used for decades, but unlike Excel, they are very flexible because they don’t require a specific application. Since they are just Values, Separated by Commas, it is a lot easier to convert the commas to an array, with each record being one line of CSV.
• XLSX – Data Import: Since Excel was released, the file type is for Excel has been XLS. However, in 2007, Microsoft created the standard of XLSX and the standard of ‘Office Open XML’. Since XLSX is now the standard, we are able to import that data into a system that supports it or open it in Text Editor and Process the vaLues.

Access

• JWT – JSON Web Token pronounced “jot”, is a type of authentication based on JSON encryption. The tokens are signed either using a private secret or a public/private key. It is used as part of an OAuth process, but requires a lot more information called claims. JWTs can be used with Web Applications, SSO, etc. This is an option but we try to stay away, but you may hear it mentioned.
• OAuth – This term refers to a particular type of authentication that can be used when making or receiving web requests. It is more secure than an API Key or Basic Auth, and requires more information to setup.
• API Key – This term refers to a particular type of authentication that uses a uniquely provided “password” that allows certain access.
• Basic Authentication – This term refers to a particular type of authentication that uses a username and password combination that correlates to a user account.

Concepts

• CRUD – When you are making changes via Create / Read / Update / Delete, if you make a mistake, you might say CRUD.
• HTTP Methods (Verbs) –
  • POST Action – Most of the time it is Create. However, sometimes with an Upsert, it will Update depending on how the system is setup.
  • GET method – Visiting a website from a Browser is a ‘GET’, Importing Data from another source into your environment
  • POST vs PUT – Think about it as you originally send a letter to the POST Office for the process to start. Once the mail person gets to your House, it is an Update to their Delivery on the Route. A POST is to Create the Record and a Put is to Update a Specific Record.
  • Delete Method – Trick Question: Delete is Delete
• Uni-directional or One-way – This term describes an integration where data only moves between systems A and B in one direction (either B to A or A to B, but not both).
• Bi-directional or Two-way – This term describes an integration where data moves between systems A and B in both directions (A gets data from B, and B gets data from A). Bi-directional integrations are usually more complex to build than Unidirectional integrations.
• Push – A Push is an Integration architecture where a Secondary Server can ‘Push’ or Send data to another System. The Push can be done on a schedule or on an action taking place (hitting submit or values changing). Push can also use Event Driven
• Pull – A Pull is where the System you are trying to bring the data into, is going out to get the data from the Server. You are PULLING that data down to your environment.
• Polling or Interrogation– Polling is the concept of always Pulling or asking the server if there are any data available to be processed, but on a consistent basis. This will matter if you have no ability to send the data from the other system. However, more polling to more devices means more bandwidth and more data. Poll people Polling systems!
• Data/Body – Depending on what data is required and optional, you can provide different sets of data. This can be a JSON Object, JSON String, Query String, or Binary like an Image.
• HTTP Request – HTTP Request is an Internet Standard for the Application Layer to implement. That means that a program like Postman, or cUrl is able to make HTTP Requests on behalf of the user. Since it is a Protocol, there are only implementations of the Protocol on an Application. While this can take place on a Web Browser, most people associate the World Wide Web/Internet to HTTP. However, there are other programs that can benefit from HTTP Requests, and enhance the functionality.
• cUrl – Client For URL’ (cUrl) is a Command-line service that provides the ability to connect to another system, but from a Server where additional functionality is possible. It uses HTTP Requests to execute calls.
• API – Application Programming Interface (API) is a way for two or more computer programs to communicate with each through some means. The API documentation is used to describe how you can use each piece of the integration. Let’s say you bought your second desk from IKEA. Since you get the basic concept of building the desk from scratch, you can start with building instead of the documentation. If you then do need the documentation, it is there to provide you with the Methods, Endpoints, Options, and Body.
• Interact with REST API Integrations – Postman, cUrl, ServiceNow, PHP, Python, etc

Issues arising

• Bad or inconsistent data – Many customers like to make claims about their data using words like “always” and “never”, but the reality is that “always” and “never” often mean “mostly” and “only sometimes”. Sometimes these differences can cause problems down the line when they are discovered late in development. It’s good to get a look at sample data as early as possible, and to make it clear to the client that they are expected to provide correct data to the greatest extent possible
• Waiting on credentials – It’s important that we get the access we need early on. Sometimes it takes time to provision credentials for a service account, so the earlier we can identify what access we will need and initiate that process, the better
• Poor documentation – When talking about integrating with a 3rd-party system, it’s important to identify how much documentation is available early on. Poorly documented systems/APIs will need to be given additional time in scoping.