Add Cloudflare IP range importer

[fershad]

Fixes #569

This PR creates a new Importer for Cloudflare's public IP ranges. It fetches data from two text files which reflect the public IPv4 and IPv6 ranges that Cloudflare maintain.

TXT IPv4 list: https://www.cloudflare.com/ips-v4/#
TXT IPv6 list: https://www.cloudflare.com/ips-v6/#

[gitpod-io]

[fershad]

@mrchrisadams I think this PR is ready for review. I'm not sure how it can be tested or verified though. Will take your lead.

[mrchrisadams]

hi @fershad ! If the importer works by fetching data, and then importing it, we could check by writing a test to run the importer and see if the provider, has the new IP ranges added. Are you cool with me adding a test along these lines to this PR?

[fershad]

Thanks @mrchrisadams. Yes that kind of test does make sense. Please do add it, it would be a good reference to have for future importers as well.

[github-actions]

Old Energy Estimation

Eco-CI Output:

Label	🖥 avg. CPU utilization [%]	🔋 Total Energy [Joules]	🔌 avg. Power [Watts]	Duration [Seconds]
Total Run (incl. overhead)	25.8505	464.147	3.84	121
checkout	8.77	3.6079	3.61	1
pip install uv wheel	8.88	3.59008	1.80	2
pip install requirements	70.9933	40.1041	5.73	7
pytest	23.6341	416.845	3.82	109

🌳 CO2 Data:
City: Boydton, Lat: 36.6534, Lon: -78.375
IP: 52.167.166.88
CO₂ from energy is: 0.186587094 g
CO₂ from manufacturing (embodied carbon) is: 0.034522971 g
Carbon Intensity for this location: 402 gCO₂eq/kWh
SCI: 0.221110 gCO₂eq / pipeline run emitted

[github-actions]

Old Energy Estimation

Eco-CI Output:

Label	🖥 avg. CPU utilization [%]	🔋 Total Energy [Joules]	🔌 avg. Power [Watts]	Duration [Seconds]
Total Run (incl. overhead)	26.0133	465.246	3.75	124
checkout	8.02	3.56159	1.78	2
pip install uv wheel	8.12	3.56308	1.78	2
pip install requirements	67.2167	38.7752	5.54	7
pytest	23.5678	419.346	3.81	110

🌳 CO2 Data:
City: San Jose, Lat: 37.1835, Lon: -121.7714
IP: 52.238.27.78
CO₂ from energy is: 0.077230836 g
CO₂ from manufacturing (embodied carbon) is: 0.035378912 g
Carbon Intensity for this location: 166 gCO₂eq/kWh
SCI: 0.112610 gCO₂eq / pipeline run emitted

[mrchrisadams]

Hey @fershad - I've added:

1. tests exercising this code
2. a management command so we can run this on the command line with update_networks_in_db_cloudflare
3. some cleanup of the implementation code
4. updates to the cronjob that would run this. For now the job is commented out though.

[github-actions]

Old Energy Estimation

Eco-CI Output:

Label	🖥 avg. CPU utilization [%]	🔋 Total Energy [Joules]	🔌 avg. Power [Watts]	Duration [Seconds]
Total Run (incl. overhead)	28.2749	438.392	3.95	111
checkout	13.23	3.74373	1.87	2
pip install uv wheel	13.34	3.74414	3.74	1
pip install requirements	68.8867	39.3387	4.92	8
pytest	25.8826	391.565	4.00	98

🌳 CO2 Data:
City: Washington, Lat: 38.7095, Lon: -78.1539
IP: 40.76.238.208
CO₂ from energy is: 0.174041624 g
CO₂ from manufacturing (embodied carbon) is: 0.031669833 g
Carbon Intensity for this location: 397 gCO₂eq/kWh
SCI: 0.205711 gCO₂eq / pipeline run emitted

[mrchrisadams]

hi Fish! Thanks for making this! I can confirm this adds the IP ranges listed at the endpoints you listed.

I think this is good to merge in, but we'll likely need to make a change later to actually switch this on as I think it make sense to make sure we have a named technical contact to speak to first before running it on the regular.

[mrchrisadams]

this is the command we would run on weekly import like we do with AWS, GCP, and Microsoft

[github-actions]

Eco-CI Output:

Label	🖥 avg. CPU utilization [%]	🔋 Total Energy [Joules]	🔌 avg. Power [Watts]	Duration [Seconds]
Total Run (incl. overhead)	22.9241	511.045	3.62	141
checkout	10.8	3.67623	1.84	2
pip install uv wheel	10.87	3.66029	3.66	1
pip install requirements	69.0683	39.6261	5.66	7
pytest	20.8754	464.082	3.63	128

🌳 CO2 Data:
City: Phoenix, Lat: 33.4475, Lon: -112.0866
IP: 20.171.122.191
CO₂ from energy is: 0.258077725 g
CO₂ from manufacturing (embodied carbon) is: 0.040229247 g
Carbon Intensity for this location: 505 gCO₂eq/kWh
SCI: 0.298307 gCO₂eq / pipeline run emitted

[timcowlishaw]

Hey @mrchrisadams @fershad FYI, i'm gonna take over this branch and get it over the line! Warning you as am about to force-push a mahooosive rebase :-)

[timcowlishaw]

Deployment checklist:

• Add CLOUDFLARE_PROVIDER_ID, CLOUDFLARE_REMOTE_API_ENDPOINT_IPV4, and CLOUDFLARE_REMOTE_API_ENDPOINT_IPV6` env variables
• Run setup_cronjobs ansible playbook

[timcowlishaw]

(Added the extra env variable plumbing in the github deploy pipeline and ansible template)