All systems operational
Refreshed
Components
Asia East Servers
Operational
Uptime 100.000 % Nov 13, 2024
Uptime 100.000 % Nov 14, 2024
Uptime 100.000 % Nov 15, 2024
Uptime 100.000 % Nov 16, 2024
Uptime 100.000 % Nov 17, 2024
Uptime 100.000 % Nov 18, 2024
Uptime 100.000 % Nov 19, 2024
Uptime 100.000 % Nov 20, 2024
Uptime 100.000 % Nov 21, 2024
Uptime 100.000 % Nov 22, 2024
Uptime 100.000 % Nov 23, 2024
Uptime 100.000 % Nov 24, 2024
Uptime 100.000 % Nov 25, 2024
Uptime 100.000 % Nov 26, 2024
Uptime 100.000 % Nov 27, 2024
Uptime 100.000 % Nov 28, 2024
Uptime 100.000 % Nov 29, 2024
Uptime 100.000 % Nov 30, 2024
Uptime 100.000 % Dec 01, 2024
Uptime 100.000 % Dec 02, 2024
Uptime 100.000 % Dec 03, 2024
Uptime 100.000 % Dec 04, 2024
Uptime 100.000 % Dec 05, 2024
Uptime 100.000 % Dec 06, 2024
Uptime 100.000 % Dec 07, 2024
Uptime 100.000 % Dec 08, 2024
Uptime 100.000 % Dec 09, 2024
Uptime 100.000 % Dec 10, 2024
Uptime 100.000 % Dec 11, 2024
Uptime 100.000 %
Uptime over the past 30 days 100.000 % uptime
Incidents
Asia Southeast Servers
Operational
Uptime 100.000 % Nov 13, 2024
Uptime 100.000 % Nov 14, 2024
Uptime 100.000 % Nov 15, 2024
Uptime 100.000 % Nov 16, 2024
Uptime 100.000 % Nov 17, 2024
Uptime 100.000 % Nov 18, 2024
Uptime 100.000 % Nov 19, 2024
Uptime 100.000 % Nov 20, 2024
Uptime 100.000 % Nov 21, 2024
Uptime 100.000 % Nov 22, 2024
Uptime 100.000 % Nov 23, 2024
Uptime 100.000 % Nov 24, 2024
Uptime 100.000 % Nov 25, 2024
Uptime 100.000 % Nov 26, 2024
Uptime 100.000 % Nov 27, 2024
Uptime 100.000 % Nov 28, 2024
Uptime 100.000 % Nov 29, 2024
Uptime 100.000 % Nov 30, 2024
Uptime 100.000 % Dec 01, 2024
Uptime 100.000 % Dec 02, 2024
Uptime 100.000 % Dec 03, 2024
Uptime 100.000 % Dec 04, 2024
Uptime 100.000 % Dec 05, 2024
Uptime 100.000 % Dec 06, 2024
Uptime 100.000 % Dec 07, 2024
Uptime 100.000 % Dec 08, 2024
Uptime 100.000 % Dec 09, 2024
Uptime 99.650 % Dec 10, 2024
Uptime 100.000 % Dec 11, 2024
Uptime 100.000 %
Uptime over the past 30 days 99.988 % uptime
Incidents
Server Performance Update
Resolved | Feb 07, 2024 | 15:50 GMT+08:00
On Feb 5th we increased the CPU of the Singapore data center, which caused a downtime from 18:33:31 to 18:39:05 during the update.
On 6th of Feb morning, we experienced a downtime from 08:40:33 AM to 08:44:29 AM.
We decided to revert to the previous state - this generated a second downtime from 08:55:39 AM to 09:09:08 AM.
Since then the system is working well.
Issues with pdf exports
Resolved | Jun 26, 2023 | 09:16 GMT+08:00
Attempts to generate pdf exports containing a high number of images led to a temporary peak in the server RAM usage, resulting in interruptions:
Jun 15, 2023, 17:46:12 - 17:49:12
Jun 15, 2023, 17:57:58 - 17:59:53
Jun 15, 2023, 23:02:05 - 23:03:34
Jun 15, 2023, 23:04:42 - 23:10:00
Jun 15, 2023, 23:11:15 - 23:12:45
We have standardised the restrictions associated with PDF exports across the entire platform to prevent a recurrence of this issue.
Internal issues with pdf engine
Resolved | Jun 13, 2023 | 15:49 GMT+08:00
Errors occurred in the engine used by the system to convert uploaded pdf files to drawings, causing an interruption:
Jun 6, 2023, 10:56:33 - 11:14:00
We are investigating and monitoring these errors to prevent them from reoccurring.
Short downtimes
Resolved | May 26, 2023 | 17:28 GMT+08:00
Very high usage during the below times which made the service unavailable for some users:
Apr 25, 2023, 14:39:41 - 14:52:08
May 3, 2023, 10:22:01 - 10:33:12
May 23, 2023, 08:05:20 - 08:07:44
We have identified an excessive quantity of data synchronised by some accounts, and we are optimising the data storage to scale better.
Some errors occurred in the engine used by the system to resize images in pdf exports, causing some short interruptions:
May 5, 2023, 08:05:01 - 08:35:47
May 16, 2023, 10:28:34 - 10:39:42
May 23, 2023, 07:58:03 - 08:01:14
We are investigating and monitoring these unexpected errors in order to fix them and recover from them.
Some security updates were installed during the night, leading to brief downtimes:
May 23, 2023, 03:26:30 - 03:29:39
May 23, 2023, 06:21:46 - 06:24:02
Major outage in Asia from Azure
Resolved | Feb 08, 2023 | 04:40 GMT+08:00
Novade services are down in some parts of Asia due to a major outage of Microsoft Azure's servers.
We are actively looking into the issue and will update you as soon as we have more information.
Update from Azure:
Issue Datacenter Cooling Event - Southeast Asia
Issue Summary: Starting around 20:19 UTC on 7 February 2023, a utility power surge in the Southeast Asia region tripped all of the chiller units for one datacenter offline. While working to restore the chiller units, temperatures in the datacenter increased so we have proactively powered down four storage scale units to avoid potential damage to the hardware.
https://status.azure.com/en-us/status
While Azure's outage is still ongoing, we managed to resume services with new servers on Wednesday 8/02 at 4pm SG time
Heavy usage
Resolved | Jan 03, 2023 | 08:00 GMT+08:00
Very high usage during the below times which made the service unavailable for some users.
Dec 6, 2022, 08:18:52-08:33:46 09:00:04-09:17:17
Dec 7, 2022, 08:38:58-08:45:17 08:50:07-09:03:58
Dec 8, 2022, 06:08:29-06:18:19
Dec 13, 2022, 08:29:43-08:34:15 08:56:35-09:01:27
Dec 21, 2022, 18:56:26-19:01:19
Dec 22, 2022, 16:40:13-16:51:45
Dec 24, 2022, 07:14:46-07:32:30
Dec 28, 2022, 14:29:50-14:43:20
Jan 3, 2023, 09:05:01-09:11:20 09:48:25-09:56:44
Jan 4, 2023, 09:43:49-09:48:50
We have identified a series of concurring events and we optimised the code and the setup to scale better.
EMEA Servers
Operational
Uptime 99.402 % Nov 13, 2024
Uptime 100.000 % Nov 14, 2024
Uptime 100.000 % Nov 15, 2024
Uptime 100.000 % Nov 16, 2024
Uptime 100.000 % Nov 17, 2024
Uptime 100.000 % Nov 18, 2024
Uptime 100.000 % Nov 19, 2024
Uptime 100.000 % Nov 20, 2024
Uptime 100.000 % Nov 21, 2024
Uptime 100.000 % Nov 22, 2024
Uptime 100.000 % Nov 23, 2024
Uptime 100.000 % Nov 24, 2024
Uptime 100.000 % Nov 25, 2024
Uptime 100.000 % Nov 26, 2024
Uptime 100.000 % Nov 27, 2024
Uptime 100.000 % Nov 28, 2024
Uptime 100.000 % Nov 29, 2024
Uptime 100.000 % Nov 30, 2024
Uptime 100.000 % Dec 01, 2024
Uptime 100.000 % Dec 02, 2024
Uptime 100.000 % Dec 03, 2024
Uptime 100.000 % Dec 04, 2024
Uptime 100.000 % Dec 05, 2024
Uptime 100.000 % Dec 06, 2024
Uptime 100.000 % Dec 07, 2024
Uptime 100.000 % Dec 08, 2024
Uptime 100.000 % Dec 09, 2024
Uptime 100.000 % Dec 10, 2024
Uptime 100.000 % Dec 11, 2024
Uptime 100.000 %
Uptime over the past 30 days 99.980 % uptime
Incident history
Server Performance Update
Resolved | Feb 07, 2024 | 15:50 GMT+08:00
On Feb 5th we increased the CPU of the Singapore data center, which caused a downtime from 18:33:31 to 18:39:05 during the update.
On 6th of Feb morning, we experienced a downtime from 08:40:33 AM to 08:44:29 AM.
We decided to revert to the previous state - this generated a second downtime from 08:55:39 AM to 09:09:08 AM.
Since then the system is working well.
Affected components
- Asia Southeast Servers
Issues with pdf exports
Resolved | Jun 26, 2023 | 09:16 GMT+08:00
Attempts to generate pdf exports containing a high number of images led to a temporary peak in the server RAM usage, resulting in interruptions:
Jun 15, 2023, 17:46:12 - 17:49:12
Jun 15, 2023, 17:57:58 - 17:59:53
Jun 15, 2023, 23:02:05 - 23:03:34
Jun 15, 2023, 23:04:42 - 23:10:00
Jun 15, 2023, 23:11:15 - 23:12:45
We have standardised the restrictions associated with PDF exports across the entire platform to prevent a recurrence of this issue.
Affected components
- Asia Southeast Servers
Update of SSL certificate
Resolved | Jun 13, 2023 | 15:51 GMT+08:00
The SSL certificate was updated on the server which resulted in a downtime:
Jun 12, 2023, 16:47:00 - 17:01:00
To prevent this from reoccurring, SSL certificate update will not be done during day-shift.
Affected components
- Asia East Servers
Internal issues with pdf engine
Resolved | Jun 13, 2023 | 15:49 GMT+08:00
Errors occurred in the engine used by the system to convert uploaded pdf files to drawings, causing an interruption:
Jun 6, 2023, 10:56:33 - 11:14:00
We are investigating and monitoring these errors to prevent them from reoccurring.
Affected components
- Asia Southeast Servers
Short downtimes
Resolved | May 26, 2023 | 17:28 GMT+08:00
Very high usage during the below times which made the service unavailable for some users:
Apr 25, 2023, 14:39:41 - 14:52:08
May 3, 2023, 10:22:01 - 10:33:12
May 23, 2023, 08:05:20 - 08:07:44
We have identified an excessive quantity of data synchronised by some accounts, and we are optimising the data storage to scale better.
Some errors occurred in the engine used by the system to resize images in pdf exports, causing some short interruptions:
May 5, 2023, 08:05:01 - 08:35:47
May 16, 2023, 10:28:34 - 10:39:42
May 23, 2023, 07:58:03 - 08:01:14
We are investigating and monitoring these unexpected errors in order to fix them and recover from them.
Some security updates were installed during the night, leading to brief downtimes:
May 23, 2023, 03:26:30 - 03:29:39
May 23, 2023, 06:21:46 - 06:24:02
Affected components
- Asia Southeast Servers
Major outage in Asia from Azure
Resolved | Feb 08, 2023 | 04:40 GMT+08:00
Novade services are down in some parts of Asia due to a major outage of Microsoft Azure's servers.
We are actively looking into the issue and will update you as soon as we have more information.
Update from Azure:
Issue Datacenter Cooling Event - Southeast Asia
Issue Summary: Starting around 20:19 UTC on 7 February 2023, a utility power surge in the Southeast Asia region tripped all of the chiller units for one datacenter offline. While working to restore the chiller units, temperatures in the datacenter increased so we have proactively powered down four storage scale units to avoid potential damage to the hardware.
https://status.azure.com/en-us/status
While Azure's outage is still ongoing, we managed to resume services with new servers on Wednesday 8/02 at 4pm SG time
Affected components
- Asia Southeast Servers
Heavy usage
Resolved | Jan 03, 2023 | 08:00 GMT+08:00
Very high usage during the below times which made the service unavailable for some users.
Dec 6, 2022, 08:18:52-08:33:46 09:00:04-09:17:17
Dec 7, 2022, 08:38:58-08:45:17 08:50:07-09:03:58
Dec 8, 2022, 06:08:29-06:18:19
Dec 13, 2022, 08:29:43-08:34:15 08:56:35-09:01:27
Dec 21, 2022, 18:56:26-19:01:19
Dec 22, 2022, 16:40:13-16:51:45
Dec 24, 2022, 07:14:46-07:32:30
Dec 28, 2022, 14:29:50-14:43:20
Jan 3, 2023, 09:05:01-09:11:20 09:48:25-09:56:44
Jan 4, 2023, 09:43:49-09:48:50
We have identified a series of concurring events and we optimised the code and the setup to scale better.
Affected components
- Asia Southeast Servers