About Client
CrowdSec is a hyper growth cybersecurity company with 300k+ installations. According to G2 it’s a market leader in 8+ categories.
Problem Statement
CrowdSec offers their users an open source threat detection engine. This engine reads logs and other data sources to detect malicious IPs. Users can then use various remediation componenets provided by CrowdSec to act upon these IPs at various levels. In addition to this, the malicious IPs are sent to CrowdSec. CrowdSec then aggregates this data, applies lot of rules and provides it back to the users in the form of a IP blocklist. So users can block these IPs before even getting attacked.
One of such remediation component was for Cloudflare, which was also developed by us long time ago. This component would sync malicious IPs from CrowdSec, with a Cloudflare block list. It would create a firewall rule in Cloudflare to perform user configured actions(captcha, block, log etc) on IPs originating from this block list.
The component was one of the most used remediation components. However, with time CrowdSec’s network grew, so did the number of reported malicious IPs. This resulted in larger blocklist.
Cloudflare has a limit of 10K IPs. Thus the remediation component would fail to sync the blocklist with Cloudflare. This would result in users not being able to block malicious IPs.
Solution
We decided to build a Cloudflare Worker based solution to solve this problem.
A Cloudflare Worker is a serverless function that runs on Cloudflare’s edge network.
The remediation component would do the following:
- Create a Cloudflare Worker KV store to store the malicious IPs. Worker KV is a key value store provided by Cloudflare.
- Poll the CrowdSec API for malicious IPs and store them in the KV store.
- Deploy a Cloudflare Worker in front of the user’s web server as a “reverse proxy”. This worker would block/captcha IPs if they are present in the KV store.
We also supported remediations on Country and ASN level.
This solution was validated by Cloudflare.
Implementation
We used Go to implement the remediation component. We used the official Cloudflare SDK to implement the worker. The remediation component includes utilities to cleanup everything it creates and to generate the Cloudflare configuration in a single command.
The worker was implemented in JavaScript, it was bundled using Webpack. We implemented the captcha action in the worker using Cloudflare’s Turnstile. The remediation component manages the creation of the turnstile keys and rotates them periodically.
We also wrote integration tests to ensure the component works as expected. These use a testing Cloudflare account.
Challenges
This was smooth sailing. The most relatively challenging part was to implement the captcha action in the worker. We had to handle the cases like validating whether the user had already solved the captcha or not. We did this by issuing a signed JWT cookie to the user upon successfully solving the captcha.
Results
The new remediation component can handle 150K+ malicious IPs (that’s how much we tested for but it should be able to handle more without breaking a sweat).
The remediation component is open source and available on GitHub.
It still has the same user friedly CLI as the previous remediation component. Users can protect their web servers from malicious IPs in a two commands.