Status

Failed deploy

Date: September 19, 2023
Service Affected: Dashboard

13:15 UTC

We have now restored access, and successfully pushed the metadata hotfix. Access is fully restored.

12:10 UTC

During a planned hotfix of the platform this morning to resolve some metadata issues, we have encountered a build issue in production that has led to a failed deploy.
Unfortunately the rollback will take up to 45 minutes to build and deploy, so for the time being the Dashboard is temporarily unavailable.

Rest assured we're working as quickly as possible to restore access, and we will update once more when the deployment is complete.

Please accept our apologies for the interruption. We do not expect devices to be affected by this temporary issue, as it only affects the Dashboard.

Dashboard availability

Date: September 18, 2023
Service Affected: Dashboard

11:50 UTC 20th September

We have completed work on both the Dashboard and the Android application to fully resolve all scoped issues.

13:30 UTC

We have rolled out a hotfix to partially resolve the traffic spikes, and allowed metadata updates to flow.

We will now monitor the platform and update once more tomorrow.

12:40 UTC 19th September

We have isolated the source of traffic to the release of a device metadata feature that is not queuing updates appropriately.

For the moment we are rejecting these updates at our firewall while we undergo the necessary work to appropriately resolve the traffic spikes bottle-necking our infrastructure.

We expect to resolve this partially today, and fully before the end of the week.

Metadata updates show in Devices > Device Details > Hardware - - (battery, wifi/cellular signal, active radio) will gradually return this week.

16:30 UTC

The mambo EMM platform has been subject to greatly increased traffic over the last 24 hours, occasionally causing the dashboard to slow down or show momentary errors.

the mambo EMM team are actively monitoring and aiming to reduce traffic from unknown sources through our infrastructure providers.

If customers notice dashboard slowdown, or errors while utilising the dashboard, please reach out to support for assistance.

Devices will not be affected, as this is only concerning traffic to the mambo EMM dashboard at this time.

This incident will be updated tomorrow, UTC.

Remote Control connectivity

Date: September 15, 2023
Service Affected: Remote Control

10:00 UTC, Monday Sep 18th

We isolated the issue to an infrastructure component in our Azure backend that failed to send FCM messages (those that wake a device for a Remote Support session). We resolved the component issue and restored service late Friday UTC.

14:30 UTC

We are aware of issues initiating Remote Control sessions with managed devices.

Our team is currently investigating.

If you see "Waking Device" for a long period of time with no successful connection, please contact support.

Issue with device search

Date: August 31, 2023
Service Affected: Dashboard

13:30 UTC

We have resolved the issue and restored access to the mambo EMM dashboard.

Thank you for your patience.

13:00 UTC

We have identified the issue as an incomplete database table migration during the 1.16 release. We are undertaking emergency maintenance to resolve it between 13:15 and 13:30 UTC. The mambo EMM dashboard will be temporarily unavailable.

Your devices will be unaffected by this, as it only concerns the admin-facing mambo EMM dashboard.

08:00 UTC

We're aware of an issue currently with device search. Customers attempting to use the search box to find devices will be met with a persistent loading icon and no search results will be displayed.

We are actively investigating the issue, and have isolated it to the 1.16 release published yesterday.

For the time being search will not function. To restore access to the full device list please delete all characters from the search box. Refreshing the page alone will not work, as we persist the search query when this is done.

We apologise for the inconvenience and aim to resolve this as soon as possible.

Unscheduled system maintenance

Date: August 5, 2023
Service Affected: Dashboard

5:20 UTC

We have now resolved the issue and restored access to mambo EMM.

5:10 UTC

The cause of the issue has been identified as an uncaught error thrown during processes due to an unstabke Redis connection.

We are restoring access to the platform and verifying Redis stabiltiy is normal.

4:40 UTC

Our dashboard and backend API are experiencing stability issues and we are investigating reports of our maintenance page and/or spinning loading icon displaying for customers after successful login.

Devices are unaffected. However at this time it is not possible to use the mambo EMM web console.

We are actively investigating the issue, and have isolated it to a patch released earlier today.

Unscheduled system maintenance

Date: July 21, 2023
Service Affected: Dashboard

6:00 UTC

We have now resolved the issue and restored access to mambo EMM.

5:00 UTC

The cause of the stability issues has been identified. Due to some abnormal database entries a migration scirpt failed to run.

We are restoring access to the platform and attempting to manually sort the database entries.

20:00 UTC

We are continuing work to restore access to customer teams.

18:14 UTC

Our dashboard and backend API are experiencing stability issues and we are investigating reports of our maintenance page and/or spinning loading icon displaying for customers after successful login.

Devices are unaffected. However at this time it is not possible to use the mambo EMM web console.

We are actively investigating the issue, and have isolated it to a patch released earlier today.

Unscheduled system maintenance

Date: July 17, 2023
Service Affected: Dashboard

21:55 UTC

We have now resolved the issue and restored access to mambo EMM. Queued updates from devices are being processed with a short delay which will resolve itself within the next hour.

To mitigate the rogue traffic we have increased platform scaling capacity, and will implement a more permanent solution within the 1.16 release cycle to address any attempts to target unknown or unsupported endpoints on our API.

21:40 UTC

The cause of the stability issues has been identified. The platform has been subject to sudden, rogue API traffic attempting to hit an unsupported endpoint.

We are restoring access to the platform and attempting to mitigate the source of the traffic.

20:59 UTC

Our dashboard and backend API have reported stability issues in the last 10 minutes and we are investigating reports of our maintenance page displaying for customers.

Devices are unaffected, however while our API is rejecting traffic some updates sent from devices (heartbeat, location) will not be received at this time. They will queue and retry once the platform is fully operational.

Login issues

Date: July 6, 2023
Service Affected: Dashboard

14:20 UTC

We have now resolved the issue and restored access to mambo EMM.

As part of our 1.15 release we have made substantial changes to the billing and reporting systems within the application that have required background changes involving the database.

In Local and Staging environments we observed no issues when rolling these patches out to test environments repeatedly, however in Production we caught an edge case on a table migration which hung DB tasks undertaken after a version update.

We have manually applied the changes against the edge case, allowed the remaining migrations to complete, and the system is now back in a healthy state.

Thank you for your patience. We will continue monitoring platform stability over the next 24h but consider this incident resolved.

14:00 UTC

While we work to resolve the issue, customers may notice Azure App Services warnings in lieu of a blank loading screen and/or Under Maintenance pages.

This is due to the team taking the application fully offline to resolve an identified database issue.

Your devices remain unaffected.

13:30 UTC

We are aware of an issue with logging in to mambo EMM teams at present.

We have identified the underlying cause and are working to resolve it.

Your devices are not affected.

Enrolment and app installation issues

Date: June 21, 2023
Service Affected: External services

Status: Resovled.

Please be aware there is an ongoing issue with Google's infrastructure affecting app installs.

At present, applications set to REQUIRED FOR SETUP and/or FORCE_INSTALLED are failing to install.

As the mambo EMM Companion app is a REQUIRED FOR SETUP application we are aware devices may not be currently enrolling. Due to the close integration of our companion app into the mambo EMM solution, we are not able to alter the installation type due to risk of longer term enrolment issues.

This issue is out of the control of the mambo EMM team, however we are monitoring Google's communications closely for resolution.

We will resolve this incident on confirmation from Google of a resolution.

Frequent email alerts

Date: May 22, 2023
Service Affected: Email services

We are aware of an email notification issue affecting a small percentage of customers with outstanding invoices.

We are currently investigating the cause and will resolve as soon as possible.

In the interim, duplicate notifications may be discarded. We apologise for the inconvenience.

Billing warnings

Date: May 22, 2023
Service Affected: Dashboard

Please be aware a small percentage of customers are seeing a billing warning banner on their teams alerting them to a missing subscription, and a further warning in /billing suggesting there is a problem with the subscription.

This affects customers on our recently released manual billing tier, in which billing is handled outside of the mambo EMM console.

Your teams and plans are unaffected as this is a UI bug. We are working to resolve the front-end issue associated with this as quickly as possible.

We apologise for the inconvenience.

Zero-touch outage

Date: April 25, 2023
Service Affected: External services

Please be aware there is a zero-touch outage affecting all Android devices.

Google acknowledged the problem this morning (UTC) and are currently investigating.

We will update once service has been restored.

---

Update, 2023-04-25, 7PM UTC

Zero-touch is now once more functional.

Remote Control session instability

Date: December 16, 2022
Service Affected: Remote Control

We are currently monitoring reported remote control session instability.

Symptoms of this issue include:

1. Random remote control session disconnects (drops) - Fixed
2. Inability to reliably restart remote control sessions - Fixed
3. Server/Device unavailable messages within the mambo Dashboard RC interface - Fixed
4. Device performance issues - Fixed

These have been reported to affect devices with known, good connectivity and therefore rule out network instability as a cause.

Investigation into this issue is ongoing.

Updates

mambo version 1.12

Inability to reliably restart remote control sessions has been resolved under a companion app release as part of the 1.12 release cycle. Please ensure your estate is running the latest version of mambo Device Agent (1.4.8 or newer).

mambo version 1.13

Random remote control session disconnects (drops) are now reliably functioning in our testing. These situations may still occur with poor connectivity, however the internal errors relating to this with good connections should be resolved.

Server/Device unavailable messages in situations of good connectivity have been patched, and are undergoing testing & monitoring in our internal staging environments.

Device performance issues have been dramatically improved with changes to video resolution and frame rate on 2GB RAM (low_mem) devices, and a fix to replace a deprecated orientation monitoring service with a newer, more efficient method. This is version 1.4.11 of the EMM companion application, releasing ahead of mambo platform version 1.14. All customers will be updated to this version with no action necessary.

Further information

Respective patches may be viewed on our changelog.