aeolus (Posts by sourcehut)

[Resolved] Planned maintenance on all services

sourcehut — Tue, 22 Jul 2025 09:00:00 GMT

The maintenance is complete. All services are restored.

(10:15 UTC — Jul 22)

The maintenance window is now open. Disruptions will occur across all services.

(09:00 UTC — Jul 22)

Planned maintenance on July 22nd will cause intermittent outages: Planned maintenance starting at 09:00 UTC will affect all services, causing intermittent outages that are expected to last between 15 and 30 minutes each. Notably, chat.sr.ht will be rebooted, causing everyone to be disconnected from the bouncer. Most clients will reconnect automatically. The maintenance is expected to take up to a few hours.

(07:30 UTC — Jul 2)

Planned maintenance on all services

sourcehut — Tue, 22 Jul 2025 09:00:00 GMT

[Resolved] Planned bouncer ugrade on Thursday

sourcehut — Thu, 03 Apr 2025 12:00:00 GMT

Restart completed. Again, note that some upstream connections may take a while to be re-established due to rate limits.

(12:05 UTC — Apr 3)

Planned bouncer upgrade is scheduled for Thursday, April 3rd. On Thursday at 12:00 UTC we will be rolling out an upstream update for soju, the software that powers the chat.sr.ht bouncer. The bouncer will be restarted, causing everyone to be disconnected. Most clients will reconnect automatically. Note that some upstream connections may take a while to be re-established due to rate limits.

(09:55 UTC — Mar 31)

[Resolved] LLM crawlers continue to DDoS SourceHut

sourcehut — Mon, 17 Mar 2025 08:30:00 GMT

We have deployed Anubis to git.sr.ht. After some internal discussions we have ultimately decided that the best course of action to protect git.sr.ht from LLM crawlers is to deploy Anubis. This software presents some users with a proof-of-work challenge which is solved by the user’s browser with JavaScript.

This challenge is automatically bypassed for logged-in users. If your browser does not support JavaScript (or you do not wish to enable it for any other reason), log in at meta.sr.ht to circumvent it.

Note that Anubis is only being used for the web frontend. API access and git operations are unaffected.

This solution is robust and reliable. We do not want to leave it enabled indefinitely, but considering that the user impact is minimal and it is sufficient to mitigate the LLM traffic, we consider the matter closed and are closing this notice. Thank you for your patience while we prepared our mitigations.

(09:00 UTC — Mar 24)

SourceHut continues to face disruptions due to aggressive LLM crawlers. We are continuously working to deploy mitigations. We have deployed a number of mitigations which are keeping the problem contained for now. However, some of our mitigations may impact end-users.

In particular, we have deployed Nepenthes to certain routes which are associated with large volumes of LLM-related traffic. You may encounter certain pages which are not usable as a result, especially if you are not logged in.

Mitigations only affect the web frontend of SourceHut: SSH access, git operations, API access, and so on, should behave normally.

We understand that some of our mitigations are user-impacting. We apologize for the inconvenience. These measures are temporary, but we do not have an estimate for when they will no longer be required. To be honest, we are running out of ideas for how to deal with these LLM bots. Your patience is appreciated.

If you are having problems using the SourceHut web UI:

First, log into your SourceHut account. Logged-in users bypass most of our mitigations. If that does not work, please contact support on IRC or via email.

If your cloud server is unable to reach SourceHut:

We have unilaterally blocked several cloud providers, including GCP and Azure, for the high volumes of bot traffic originating from their networks. If your cloud server is experiencing problems using SourceHut, and you have a legitimate reason to do so, you must email support to request an exception. Please explain your use-case and include a list of affected IPs and/or subnets.

We kindly ask the administrators of SourceHut integrations to program their software with responsible usage patterns. If possible, we request that you prefer webhooks over polling for updates. If your integration performs git operations, please prefer to use git fetch to update a persistent repository, or use a shallow git clone, rather than performing a fresh clone each time your automation runs. We also request that you set a User-Agent string for your traffic which identifies your software and includes an email address that we can contact with questions and feedback, as well as clearly identifying your traffic as non-malicious so we do not mistakenly apply mitigations to you.

If you are using git(1) for git operations, you can set a User-Agent by setting the GIT_HTTP_USER_AGENT environment varaible accordignly.

If you would like advice on making your integration more efficient, or setting up webhooks, please contact support for assistance.

(08:30 UTC — Mar 17)

[Resolved] Planned maintenance on Tuesday

sourcehut — Tue, 11 Feb 2025 10:00:00 GMT

Maintenance completed. Access to all services is restored. (11:40 UTC — Feb 11)

Maintenance will take longer as expected. Due to unforeseen circumstances, the maintenance will take longer than anticipated. We are working to resolve the issue as quickly as possible. (11:00 UTC — Feb 11)

Maintenance is underway. The maintenance window is now open. (10:00 UTC — Feb 11)

Planned maintenance on February 11th will cause intermittent outages: Planned maintenance starting at 10AM UTC will affect all services, causing intermittent outages that are expected to last between 15 and 30 minutes at most. Notably, chat.sr.ht will be rebooted, causing everyone to be disconnected from the bouncer. Most clients will reconnect automatically. The maintenance is expected to take no more than an hour.

Have a nice weekend! (10:50 UTC — Feb 6)

[Resolved] LLM crawlers continue to DDoS SourceHut

sourcehut — Thu, 23 Jan 2025 09:30:00 GMT

SourceHut continues to face disruptions due to aggressive LLM crawlers. We are continuously working to deploy mitigations, instead of doing any of the other numerous things we have to do, like develop new features, upgrade our infrastructure, or respond to support emails in a timely manner.

We have deployed a number of mitigations which are keeping the problem contained for now. However, some of our mitigations may impact end-users.

If you are unable to use the SourceHut web UI:

First, log into your SourceHut account. Logged-in users bypass some of our mitigations. If that does not work, please contact support on IRC or via email.

If your cloud server is unable to reach SourceHut:

We have unilaterally blocked several cloud providers, including GCP and Azure, for the high volumes of bot traffic originating from their networks. If you have a legitimate reason to access SourceHut from these networks, you must email support to request an exception. Explain your use-case and provide the affected IP addresses.

Note that our mitigations targetting cloud servers affect git operations.

(09:30 UTC — Jan 23)

[Resolved] Ongoing DDoS of git.sr.ht and hg.sr.ht

sourcehut — Thu, 02 Jan 2025 19:30:00 GMT

Mitigations completed: We believe that we have the attack under control and that there is no need for end-user impacting mitigations to remain in place. (09:00 UTC — Jan 3)

Attack still ongoing, but under control: Good morning. The attack is still ongoing, but we have been refining our mitigations this morning. We addressed some cases which were causing git and hg to crash under load, and narrowed our mitigations to better characterize the load. All of our mitigations have now been disabled for logged-in users, reducing the impact on legitimate traffic – if you hit an error page, log in and try again.

Thank you for your patience. (09:00 UTC — Jan 3)

A distributed denial of service (DDoS) attack on git.sr.ht is underway: An attack similar to the one we encountered in December is once again underway, summoning our sysadmins out of their holiday plans. We have deployed mitigations which have mostly restored service but there are some user-facing impacts for the time being.

Happy new year! (19:30 UTC — Jan 2)

[Resolved] Ongoing DDoS of git.sr.ht

sourcehut — Wed, 04 Dec 2024 12:30:00 GMT

Mitigations complete. We have finished mitigating the attack and have removed user-impacting mitigations. (17:00 UTC — Dec 4)

Attack partially mitigated. We have partially mitigated the attack to restore service to git.sr.ht, but many features of the web UI are currently disabled to reduce load. (12:53 UTC — Dec 4)

A distributed denial of service (DDoS) attack on git.sr.ht is underway: A denial of service attack against git.sr.ht is currently underway. We are working on deploying a mitigation. (12:42 UTC — Dec 4)

[Resolved] Planned maintenance on Monday

sourcehut — Mon, 25 Nov 2024 13:00:00 GMT

Maintenance work is completed. All services should be operating normally. Please let us know if you notice any lingering problems. (13:26 UTC — Nov 25)

The maintenance window is now open. (13:00 UTC — Nov 25)

Maintenance window rescheduled for 13:00 UTC. We have advanced the schedule by one hour to 13:00 UTC. (08:00 UTC — Nov 25)

Planned maintenance is scheduled for Monday, November 25th. On Monday at 13:00 UTC we will be rolling out some changes which will cause spurious errors and issues across services for the duration of the roll-out. Notably, chat.sr.ht will be rebooted, causing everyone to be disconnected from the bouncer. Most clients will reconnect automatically. The maintenance is expected to take no more than an hour.

Have a nice weekend! (14:00 UTC — Nov 22)

[Resolved] Upstream network outage

sourcehut — Wed, 30 Oct 2024 11:30:00 GMT

Upstream network issues appear resolved. We still know little of the cause but we know it broadly affected numerous providers in our service area. (14:00 UTC — Oct 30)

Upstream network issues are causing intermittent availability problems for SourceHut. An issue with our upstream network provider has affected user access to SourceHut. We do not have any further details or an ETA on a solution at this time. We expect the network to be intermittent until the upstream issue is resolved. (11:30 UTC — Oct 30)