Under Investigation 07/08/2025 - Network Interruptions NL

GreatMarko said:
Ooo... slightly off-topic, but does that mean that NVMe VPS availability will finally be coming to the EU data center?!
Click to expand...
Everything will be NVMe based yes, but I won't have the Ryzen / High Frequency CPUs. They will be much beefier EPYCs however.

TLDR, Ryzen's are *very* finnicky to get right, and we have a good recipe for them in GA , but even in WA they've been a bit of a struggle. When they perform they perform well, when they have issues they are nightmare beasts, so we need to keep them where direct staff can handle any issues that arise.
 
Good morning all Europeans, we're all waking up today with calls/messages/tickets/e-mails from our clients asking at first why they don't see their e-mails and some minutes later, asking why their websites are down. The thing is, at least on my VPS, the problem exists the last week with httpd FAILS, spamd FAILS and server restarts. I think it is related with high A.I. bots traffic these issues. What KH should do? When the service is up and running, to set a common rule and all VPS/Dedicated/Reseller clients and their services to bypass A.I. bots and get back their real web traffic. Then all the resources will come back to normal and most of the time we'll be in the realm of KH services from the past.

Now, I expect then everything comes back online to be able to get at least a partial refund or time extension of our services as bonus for the time we wasted on calls/replies to our clients since early morning today.

Let's see how long we'll be here discussing this issue.
 
alex27 said:
Is this issue only occurring with our KnownHost NL server, or is it also happening with other hosting providers?
Click to expand...
I'd be unable to comment on other providers, but I can state that anyone using PacketFabric /Unitas / INAP transit in Amsterdam, and/or any transit fed by those providers from Equinix AMS1 would be impacted.
 
VAGDesign said:
Good morning all Europeans, we're all waking up today with calls/messages/tickets/e-mails from our clients asking at first why they don't see their e-mails and some minutes later, asking why their websites are down. The thing is, at least on my VPS, the problem exists the last week with httpd FAILS, spamd FAILS and server restarts. I think it is related with high A.I. bots traffic these issues. What KH should do? When the service is up and running, to set a common rule and all VPS/Dedicated/Reseller clients and their services to bypass A.I. bots and get back their real web traffic. Then all the resources will come back to normal and most of the time we'll be in the realm of KH services from the past.

Now, I expect then everything comes back online to be able to get at least a partial refund or time extension of our services as bonus for the time we wasted on calls/replies to our clients since early morning today.

Let's see how long we'll be here discussing this issue.
Click to expand...
A.I. Bots are the plague these days. It's very difficult to block them at the server level because providers like AWS, Google Cloud and Azure have such large IP ranges and rent them servers allowing them unfettered access. If we block Google Cloud ASN IP ranges from your server, we run the risk of blocking actual services from Google.

RE: Credit, once service is restored please open a ticket with our billing department and we'll take care of you.
 
digitalboxgr said:
3 hrs downtime. In 2025. On one of the most reliable hosting providers in the world.

Like we are in the 90s.

Disappointed.
Click to expand...
I'm right there with you. Our contracts are for fully redundant circuits on diverse equipment with diverse fiber paths... yet a single location (Equinix AMS1) has an issue and we're dead in the water.
 
rajivkrchugh said:
If a similar incident occurs again tomorrow, will our setup have adequate redundancy in place to prevent service disruption?
Click to expand...
In a days time, no. the upgrades we have in motion took most of this year to put together for them to take place in a few weeks. It takes 60~ days for most vendors to build out and deploy new circuits.
 
KH-DanielP said:
I'd be unable to comment on other providers, but I can state that anyone using PacketFabric /Unitas / INAP transit in Amsterdam, and/or any transit fed by those providers from Equinix AMS1 would be impacted.
Click to expand...
Alright, could you please share the link provided by the providers you’re referring to, where they display their current update status? I assume they should have it available on their website as well — right?

That way, we can share it with our client. Since you mentioned you don’t have control over it, at least we can stay informed through those providers.
 
alex27 said:
Alright, could you please share the link provided by the providers you’re referring to, where they display their current update status? I assume they should have it available on their website as well — right?

That way, we can share it with our client. Since you mentioned you don’t have control over it, at least we can stay informed through those providers.
Click to expand...
I'd love to, except packetfabric.com (who acquired Unitas Global) doesn't publish status pages.
 
KH-DanielP said:
It's very possible it's isolated to the Packet Fabric presence within Equinix AMS1, anything there would be referenced by status.equinix.com which is locked behind their portal and only displayed as customer specific dependent upon your services with them.

I have access to it, but not to any AMS1 details for Equinix.
Click to expand...
Any Update please?
 
KH-DanielP said:
A.I. Bots are the plague these days. It's very difficult to block them at the server level because providers like AWS, Google Cloud and Azure have such large IP ranges and rent them servers allowing them unfettered access. If we block Google Cloud ASN IP ranges from your server, we run the risk of blocking actual services from Google.

RE: Credit, once service is restored please open a ticket with our billing department and we'll take care of you.
Click to expand...
Daniel, I think there must be a path of A.I. bots detection. Something that can trigger a filter and recognize if the IP belongs to a bot or not. If there isn't anything like this, we/they should invent and create it.
If Google and others merge their services with their A.I. tools, they and we are doomed.

Also, web hosting companies should have backup plans on disasters. E.g, what happens if a lunatic blows up a datacenter? Imagine all local backups and servers to be destroyed. Don't expect clients to be prepared with cloud backups to other services and in a matter of hours to move their entire client base to another company. There should be a plan for such scenarios and at a small fee, whoever wants to have that, to be able to access their data. Just a thought.
 
VAGDesign said:
Daniel, I think there must be a path of A.I. bots detection. Something that can trigger a filter and recognize if the IP belongs to a bot or not. If there isn't anything like this, we/they should invent and create it.
If Google and others merge their services with their A.I. tools, they and we are doomed.
Click to expand...
TBH, Cloudflare is at the fore-front of this, and does decent-ish. They are supposedly rolling out features that will require bots to pay you (the site owner) for access to crawl your content.


VAGDesign said:
Also, web hosting companies should have backup plans on disasters. E.g, what happens if a lunatic blows up a datacenter? Imagine all local backups and servers to be destroyed. Don't expect clients to be prepared with cloud backups to other services and in a matter of hours to move their entire client base to another company. There should be a plan for such scenarios and at a small fee, whoever wants to have that, to be able to access their data. Just a thought.
Click to expand...

I don't disagree, and we do offer various options such as remote backups stored in different facilities, but it's not possible for a small fee. To have services of that scale for say an entire facility, factoring that Hardware these days takes 6-12 weeks to get delivered. To make it practical you'd need to have enough hot-spares of hardware to take the full facility load.

We do have customers who will for example purchase two VPSs, one as a hot spare, but keep the hotspare downgraded to the lowest package level, so in this instance it could be upgraded to handle live traffic.
 
Hi Daniel, our website (www.greenheartuae.com) has been down since this morning. I saw this forum about an issue affecting a specific region, but I’m not sure if our server is located there, so I can’t tell if the two are related. Could you help clarify?
 
You must log in or register to reply here.
Top