Hello everyone.
For the past 48 hours OVH has blocked my performance hosting (cluster 107). Usually when this happens it’s because the website was hacked, but you can still access FTP to retrieve the files, find the vulnerability, fix it, and reactivate the site with a chmod 755 from Filezilla. In my case it’s different:
I no longer have FTP access: authentication works, but the permissions on the root directory (/) are set to 000, so I can’t reach my files.
I also can’t access the snap‑1, snap‑2… file archives via FTP: the permissions are also denied on the / folder.
I haven’t received any emails from OVH informing me of the site block, which normally happens when a hack occurs: you normally get an email saying “your site has been hacked, … and blocked…”. In my case there was no email, and support didn’t tell me the site had been hacked.
Support isn’t giving me any information: they told me they launched a hack scan on my files the first day, but since then there’s been no update, so they haven’t found anything. I don’t know what the problem is, they aren’t giving me back access to my snap‑X archives, they don’t tell me if they’re still there, so I don’t know if I’ve lost everything.
When I look at OVH’s maintenance activities, I see they started maintenance tasks three days ago on several clusters, including 107 (mine): https://web-cloud.status-ovhcloud.com/incidents/ywctn0lhyrjj
I don’t know if it’s a coincidence, but 24 hours later my site was blocked, my / directory was inaccessible, and I no longer have any archives.
We are following up on your report of unavailability of your hosting.
I assure you that no files have been deleted on your hosting.
Indeed, the interruption you are experiencing is directly related to this maintenance (https://web‑cloud.status‑ovhcloud.com/incidents/ywctn0lhyrjj).
This maintenance operation impacts every file UID of your hosting. Because of the very large number of files present on your hosting (more than 170 million files), the processing takes longer than expected and is therefore still ongoing.
We are aware of the inconvenience this causes and we present our sincerest apologies.
They deleted the 170 M files that were session files in a folder, but they still haven’t reactivated my hosting, which is marked as blocked. I haven’t received any email after the blockage; they gave me no explanation of what was done by the maintenance task, nor any date for bringing it back online—it’s just lamentable.
So basically they have crashed my hosting for now six days, and still haven’t done anything to bring it back online. I’m surprised to be the only one in this case, and if I really am the only one, why aren’t they focusing on my hosting to reactivate it?
I would appreciate your opinion, or testimonies from people in the same situation as me.
Due to the very large number of files present on your hosting (over 170 million files), the processing requires more time than expected and therefore remains ongoing.
You have 170 million files.
Which subscription do you have exactly with OVH?
More than 170 million files in your hosting is not a number the system can process in the usual time
The fact that you are not receiving new information does not mean you cannot request it. Reply to your support ticket and mention the incident number ywctn0lhyrjj and the cluster 107. Explain that your hosting has been blocked for 6 days, that you saw the maintenance officially finished on August 7, and that you need to know:
If the maintenance has finished and what the current status of your hosting is.
What the action plan is to reactivate your service and restore access to your files.
An estimated time for resolution.
I hope this information helps you better understand the situation and know where to push to get a response.
I hope they provide you with a solution as soon as possible.
I don’t have 170 M files, I only have 135 GB of storage used, so that’s impossible, that’s what support tells me. They say they’re session files, which seems impossible to me. I told them to delete those files.
The maintenance task isn’t finished. I’ve already asked all these questions, and I’ve received no response from support—they don’t reply. They’ve acknowledged that the fault is on their side but do nothing to restore the service.
I have a performance hosting plan. I’ve already called 1007; you can’t reach them for technical incidents anymore, they cut the line this week (I guess they had too many calls). I tried the chat, without success. I’ll try on Twitter tomorrow, thanks for the info.
Is your service still blocked?
The maintenance you reported indeed resulted in a suspension of the affected clients while it was carried out, and we have had some cases where the number of files present exceeded reasonable usage.
I have personally intervened on some cases reported by support, including one on the same cluster as yours with a profile similar to what you describe; there were indeed 170 million session files.
Which is not normal in standard usage, as session files are temporary. This indicates a defect in a plugin of your CMS or the CMS itself that does not perform regular cleanup of these files.
If your problem is still ongoing, give me your support ticket number so I can check.
--
Bruno B.
Team Lead Infrastructures Hébergements Mutualisés
Thank you for your reply Bruno. The hosting came back online yesterday, but I still don't have access to the FTP backups.
Even though there is a CMS bug that doesn't delete session files, it is not normal to initiate a cluster change (since that is what lies behind the maintenance operation without ever being explained) without having checked how many files needed to be transferred from the new to the old, and without notifying the clients either. A simple file count could have raised the issue early on: I could then have performed the deletion of the session files. And if there really were 170 M of files, I'm surprised that their deletion took 5 days as support says, that's relatively slow...
It is also not normal that support gives no information about the reasons for the hosting blockage when we ask the question and that we have to fight to obtain a semblance of an answer;
The maintenance was announced well in advance via the status page, as we always do for maintenances with known impact like this one.
And I want to make clear that no cluster change was initiated; as support told you, we had to change the UID of some hosting accounts, nothing more.
What surprised us were the few cases like yours; we know of some intensive usage, but such extremes are rare and hard to imagine for this type of product.
When we got the green light in the ticket to clean up the files, we did so, but a file manipulation triggers multiple calls to the file servers to be processed, multiplied by 170 M, which takes an enormous amount of time that we cannot compress.
For backup access, you can start the restore from the manager without issue. However, in cases like yours, copying the 170 M files is very likely to exceed the expected time limits and fail.
Via FTP you will have access to the backups made after the intervention; for those from before, they are on the old UID and a backup snapshot is immutable, so we cannot change the UID to provide access to pre‑intervention data. Only the restore via the manager handles UID adaptation during restore (note that the manager restore restores 100 % of the files, overwriting existing ones).
--
Bruno B.
Team Lead, Shared Hosting Infrastructure
Indeed, the CMS did not delete those files, I was not aware of them, but they have no impact on my clients and it concerns ONLY ME.
That is not the case with the maintenance operation that took my clients' site down for 7 days because nobody at OVH thought to check the number of files before launching it: it is not me who is behind that maintenance operation and therefore the outage of my hosting.
If the site had been properly monitored and managed, and therefore all temporary files had been deleted, you wouldn't have experienced any outage or site unavailability.
Were these 170 M files all stored in a single directory? Or in a hierarchy?
example 1: /sessions/session000000001 up to /sessions/session170012345
example 2: /sessions/000/000/session000000001 up to /sessions/170/012/session170012345
If you look at example 2, each folder contains at most 1,000 subfolders or 1,000 session files.
However, in example 1, a single folder that contains an incredibly high number of files will generate an exponential load on the server. Filesystem design will never favor this poor usage.
