Meraki drop the ball and delete customer data

[geezersoft]

Dear Meraki Customer,

 

We are reaching out to inform of you of an issue that may have impacted your Dashboard Account, including our recommended next steps for remediation.

 

On August 3rd, 2017, our engineering team made a configuration change that applied an erroneous policy to our North American object storage service and caused certain data uploaded prior to 11:20AM Pacific Time on August 3 to be deleted. The issue has since been remediated and is no longer occurring. Although no network configuration state was impacted, some uploaded images and other files may have been lost, including:

 

Dashboard

Meraki-hosted custom splash themes - in some cases, splash pages may not be fully functional (e.g. missing custom logos)

Floor plans

Dashboard branding logos (summary reports & overall navigation menu)

User-uploaded AP installation images

Systems Manager (SM)

Contact images

User-uploaded apps

Phone (MC)

Voicemail greetings

IVR menus

Hold music

Customer-uploaded contact images

Documentation for the affected features can be found here, and our engineering team has launched a page in Dashboard to help identify specific data that has been lost or that may require remediation. Additionally, Meraki is actively developing tools to make restoring files easier. We will communicate an update by the end of the day on Monday, August 7, with the current status these tools.

 

You are also welcome to open a Support case by emailing [email protected] or calling our Support hotline; one of our Support representatives would be happy to provide further detail and assist in follow-ups. If you are calling us from North America, you can reach us on the phone at 415-432-1203 or regionally from the numbers listed on https://meraki.cisco.com/support/.

 

We deeply regret and apologize for the inconvenience this has caused. Although Meraki's object storage service replicates all customer-uploaded data across multiple locations for durability, this incident revealed the administrative interface for that service to be a single point of failure. Meraki's engineers are currently working to remove that weakness going forward. In contrast, the Meraki stores network configuration data in multiple independent systems with no single administrative interface. Network configuration data is thus safe from losses of this type.

[FN-GM]

Take it they have no backups?

[geezersoft]

Reply to my help desk ticket:

 

I can understand that at the moment we are in a very crucial situation with many consequences that are pain in the neck for the administration of a dashboard network. Manually re-adding all of the missing information to your dashboard seems to be a tedious procedure and it really is. I can totally understand that since I have been to your place.

 

Our engineering team will be working over the weekend to investigate what data we can recover, as well as what tools we can build to help our customers specifically identify what has been lost from their organization. We recommend waiting until we make these tools available prior to restoring files as we will be trying to design our tools to help our customers save time. We will communicate an update by the end of the day on Monday August 7 with the current status of what resources we will be making available to help restore functionality.

 

Please accept my apologies for any inconvenience is caused and give us the time to make it work for you or at least ease the impact of the issue.

 

Thank you,

[forkies]

Did you receive this information by email, I haven't seen anything so assume it is only some customers or locations affected?

[Blue_Cookeh]

Last I heard it was customers with data held in North America.

[geezersoft]

Yes it is customers with data held in NA. I will be getting the data moved to the EU cloud next week.