One of the methods of abstraction is to integrate another service. But can you trust someone else's service? Third-party APIs are notorious for failing at the worst possible time. Over time, I ended up with a checklist. It helps a lot.
If your integration is business critical then the checklist is not optional, you need to adhere to it. If your product is in its infancy then you can take decisions as per your bandwidth and the importance of the service.
In my 5-6 years of dealing with 3rd party APIs, there are a few lessons I keep coming back to.
What if the API fails? What do you do then? This question needs to be answered and all the stakeholders should understand that this might never be an edge case. You have to have a product flow that addresses this issue. There isn't much you can do if a 3rd Party service fails, you are not in control of it.
What if the API suddenly starts returning something you've never seen before? What happens then? You have to adjust for it. Somehow.
To remain in control of the system, you need to setup an alerting flow, that tells you what failed, when it failed and why it failed, set up email alerting. But you also need to be aware of alert fatigue, too much alerting causes one to get desensitised to it. Unless it is important to receive every alert, think about putting the alerts in a tier-ing system.
Log Everything (not the sensitive data, you might get sued or the client might drop you for it), the request, the response, the time taken for the response, everything, log it. You can remove the sensitive data before logging or you can add gibberish to the sensitive data so it cannot be identified. The logs you don't collect today are the ones you'll wish you had during an incident.
SLAs, you and the service provider needs to be on the same page on this, because your system expects something and that expectation needs to be met at most times :P
What if suddenly the load increases on the 3rd party system? Will they be able to handle it? You need to ask for performance testing stats, so you can optimise your system to never overload their system.
Retries and rate limits go hand in hand. You need a retry strategy, but don't pick a number out of thin air. Agree on it with the team. The same goes for rate limiting. Do you proactively stay below the provider's limits, or do you wait until you hit a 429 and then back off? There's no universal answer. It depends on the business use case and what you're optimising for.
Understand all error messages that they send you.
Third-party integrations are not a one-time effort. APIs get new versions, response schemas evolve, and sometimes breaking changes slip through without proper communication. Stay updated with release notes and changelogs, but don't rely on them alone. If the integration is important, run automated checks against the API every day. Trust, but verify.
This is kind of my checklist, from all my experience. If the 3rd Party API is important to your system then you need to add multiple checks around it. You won't be judged on your system's most stable day. You'll be judged on the day when everything starts failing at once, and by how gracefully your system responds. It will happen on the day when you are just happy and suddenly your system starts failing.
Systems fail, that is kinda expected, but how to handle the failure tells a lot about the system and the developer. Everyone can live with happy flows.
There was a time when we weren't aware, but the 3rd party system went offline, and because of that most of our system started misbehaving, now the calls were still going out to the 3rd Party System but it was not responding. Due to this our system started lagging.
What could have been a better way to handle it?
A health check call, if the the health check call fails, you stop or at least start queueing the calls to the 3rd party system. When the health check succeeds you replay the calls. This could have prevented some confusion. Also we could have shown a system wide message that "Some things might not work as expected as System XYZ is down".
When the 3rd party API fails, you show the real message to the user, so they are aware that okay this seemed to have happened. You send out mails that the system is not behaving as expected and make all the stakeholders aware of it so there is no mass confusion and panic.
Pro active comms.
Seems okay. :')
