I
  • Status
  • Events
  • Monitors
IANRemit

powered by openstatus.dev

Back
Feb 13, 2024
3 years ago
Degraded MPESA C2B Services (STK)
IAN Remit Payment API
Resolved · February 13 at 10:33 PM (UTC) (in 4 hours)

Resolved

All Systems are back on. We appreciate all our customers for being patient with us.

Post-mortem

In light of what has happened, here are our findings:

  • API Calls to MPESA C2B started failing. The success rate of transactions dropped drastically as the MPESA API kept dropping all our connections.
  • At one point, we were completely unable to initiate any handshake with MPESA APIs through the SSL Protocol. -Initial checks suggested that MPESA had probably made some unexpected changes on their side.
  • We proceeded to upgrade our OpenSSL libraries, and even performed an entire system overhaul. None of these seemed to address the problem.
  • We then increased our number of HTTP call retries and introduced an exponential backoff factor.
  • With the aggressive retries, we noticed initial handshakes being completed and a number of transactions started to go through.
  • We then optmized our backoff factor a little bit further until we found a sweet spot.

Conclusion

We are constantly monitoring the situation. Once more, thanks again for being patient with us!

Monitoring · February 13 at 10:28 PM (UTC) (4 minutes earlier)

Progress

The system is stabilizing by the minute. We are receiving a success rate of upto 97% in the last 10 minutes.

We are closely monitoring to ensure proper resolution. We shall reach out to the MPESA in the course of the day to figure out what's happening from their end.

Monitoring · February 13 at 10:20 PM (UTC) (9 minutes earlier)

Progress

We have pushed out a number of fixes to address the MPESA C2B hitches.

We are performing multiple tests and monitoring for any failures. The MPESA C2B API seems to be more unreliable today more than ever, with a number of transactions timing out.

Identified · February 13 at 6:13 PM (UTC) (4 hours earlier)

The What

We are encountering degraded MPESA C2B services on our API.

As of this afternoon, all our HTTPS requests to the MPESA API are being closed even before they complete.

What we are doing

We are doing our best to restore services. Our team is actively implementing a fix for this.

We sincerely apologize to our clients for the inconvenience at hand.

We shall keep you posted on the situation.