Every next-generation firewall, UTM box and secure web gateway on the market can import an external list of hostnames and deny them. The work is not the import. It is the format conversion, the object ceiling, the refresh window, the block-page wording and the log line the SOC receives afterwards. This page covers all of it, with a DNS-verified list of currently-resolving phishing domains as the input.
Whatever channel delivered the lure, the click ends up as a name resolution and a connection attempt through the same box.
A phishing link can arrive by mail, by SMS forwarded to a desktop, in a chat message, on a printed conference badge as a QR code, from a search result, or from a bookmark somebody saved eight months ago when the domain still belonged to someone else. Each of those delivery paths has its own defensive product, and each of those products sees exactly one of them. The firewall or gateway sitting on the egress path sees the consequence of all of them, because the consequence is always the same: a workstation resolves a hostname and opens a connection to it. That is the property worth building on. You are not trying to guess the delivery mechanism; you are matching on the destination, which is the only part of the campaign that has to be reachable for the attack to pay off.
The distinction matters when you write the rule. A category engine answers a taxonomy question — is this site news, gambling, social media, shopping — and it answers it with a classification produced by a crawler that visited at some point. A phishing domain list answers a much narrower factual question: has this specific hostname been observed and confirmed as live credential-harvesting infrastructure? Those are not competing answers. They are different questions, and the second is the one a category engine is structurally worst at, because a look-alike sign-in page registered three days ago has no meaningful category. It lands in newly registered, uncategorised or low reputation, and the policy you have written for those buckets is where every argument on the change advisory board happens. Nobody wants to deny all uncategorised traffic. Denying a list of hostnames that have each been verified as hostile is a far easier proposition to defend.
The deny for the phishing list belongs high in the security policy, above the general permitted-web-access rule and above any allow that could shadow it, matched on the destination FQDN or domain object rather than on resolved IP addresses. Matching on IP is the mistake people make when a platform makes domain objects awkward: phishing infrastructure is overwhelmingly hosted on shared addresses, so an address-based deny either blocks nothing useful or blocks a whole hosting provider along with the customers of that provider your finance team needs to reach. Keep the match on the name.
That is because the appliance performs more than one kind of inspection. The list can back a DNS filter profile so the query never resolves, a web filter profile so the connection is refused at the proxy, and — if the appliance relays or scans mail — a rule that acts on links inside message bodies. One object, one refresh schedule, three enforcement points. That multiplication is the practical reason UTM operators tend to get more out of a domain list than a pure NGFW operator does, and it is worth checking which profiles on your platform can reference a list object before you decide where to attach it.
It shapes everything downstream on this page. A domain list is a host-level control. It can refuse a whole hostname. It cannot say “block this one page on a host that also serves ten thousand legitimate customers”, and it cannot see anything inside an encrypted session unless you have separately decided to decrypt. That is not a defect in the data; it is the resolution the enforcement point operates at. Everything below is written to that constraint rather than around it, and the companion approaches on our DNS filtering and email security pages cover the layers where a different resolution applies.
Every vendor implements the same idea and names it differently. Knowing the shared shape saves reading five sets of documentation.
The mechanism is consistent across platforms: the appliance is given a URL, it fetches a plain-text file from that URL on a timer, it parses the file into an in-memory object, and policy rules reference that object by name. PAN-OS calls it an External Dynamic List. FortiOS calls it an external threat feed or external block list. Firepower calls it a Security Intelligence feed, distributed from the management centre out to the sensors. Check Point exposes it as a custom indicator feed under Threat Prevention. Sophos exposes it as a custom web category populated from an external source. Behind those five names sits one implementation pattern, and the operational work you do is almost entirely identical regardless of which badge is on the chassis.
The constraints are narrower than people expect on their first attempt. Most platforms want one entry per line and nothing else — no CSV columns, no scheme prefix, no path component, no trailing dot, no inline comments except on lines starting with a documented comment character, and Unix line endings. A file that still has its CSV header row on line one produces either a parse failure or, more annoyingly, a list object with one junk entry in it that nobody notices for a month. Some platforms accept a leading *. for subdomain coverage while others treat the bare domain as covering subdomains implicitly; that single detail changes the size of your file by a large factor, so confirm it before you generate anything.
The URL you point the appliance at should be stable, reachable from the appliance without an egress exception nobody remembers granting, and it should answer conditional requests properly. If your staging server returns ETag and Last-Modified headers, an appliance polling more often than the file changes receives a cheap 304 rather than re-downloading and re-parsing several megabytes of hostnames. That one header pair is the difference between a polling schedule that is free and one that shows up on a graph.
ETag and Last-Modified handling turns an aggressive polling interval into a sequence of near-zero-cost 304 responses instead of repeated full downloads.Four hops, one of which you own outright. The appliances never reach the internet for this; they talk to the staging host.
Roughly thirty lines of shell, run once a day, with one sanity gate that stops a truncated download from becoming live policy.
The feed subscription delivers the whole database as CSV alongside a daily changelog of domains added and removed. For a firewall integration the changelog is useful for reporting and for anyone keeping a diff-driven update, but the conversion itself needs only the full file.
Cut the first column, normalise it, sort and deduplicate, write it out. What turns that one-liner into something you can leave unattended is the handling around it: a scratch directory nothing else reads, so a failure at any point leaves the live file untouched.
This step deserves the most attention because it protects you from your own automation. If the download is interrupted at eighty kilobytes, a naive script writes a very short list, the appliances pull it, and you have silently removed most of your coverage without a single error appearing anywhere.
If the new file has lost more than a small percentage of yesterday's entries, refuse to promote it, keep serving the old one, and raise something a human will see. Real day-to-day churn in a list of live domains is meaningful but not violent, and a sudden collapse is far more likely to be a network problem than a real event.
Rename the validated file into place so the appliance never fetches a half-written one. The rename is atomic: a device polling mid-run gets yesterday's file or today's, and never a fragment of both.
Run the conversion shortly after the rebuild lands and set appliance polling intervals independently of it. There is no benefit in twelve appliances pulling hourly from your staging host when the file changes once a day, but no harm either provided the host answers conditional requests — and a real advantage on the day you push an urgent addition of your own alongside the feed. Set the interval to your platform's cheapest refresh and let the file's modification time do the work.
#!/bin/bash
set -euo pipefail
OUT=/srv/edl
TMP=$(mktemp -d)
trap 'rm -rf "$TMP"' EXIT
# 1. Pull the current daily CSV (feed subscription, one call per day)
curl -fsS "https://phishingdetectionapi.com/api/v1/feed?apikey=$PDA_KEY" \
-o "$TMP/feed.csv"
# 2. Column 1 is the hostname. Drop the header, normalise, dedupe.
tail -n +2 "$TMP/feed.csv" \
| cut -d',' -f1 \
| tr 'A-Z' 'a-z' \
| sed 's/\.$//' \
| grep -E '^[a-z0-9.-]+\.[a-z]{2,}$' \
| sort -u > "$TMP/hosts.txt"
# 3. Sanity gate: never promote a file that lost 10% of yesterday's lines
NEW=$(wc -l < "$TMP/hosts.txt")
OLD=$(wc -l < "$OUT/phishing-full.txt" 2>/dev/null || echo 0)
if [ "$OLD" -gt 0 ] && [ "$NEW" -lt $(( OLD * 9 / 10 )) ]; then
echo "refusing short feed: $NEW lines vs $OLD yesterday" >&2
exit 1
fi
# 4. Split into parts that each fit one list object on your platform
split -l 45000 -d --additional-suffix=.txt "$TMP/hosts.txt" "$TMP/phishing-part-"
# 5. Promote atomically: the appliance never sees a partial file
cp -f "$OUT/phishing-full.txt" "$OUT/phishing-full.prev" 2>/dev/null || true
mv "$TMP/hosts.txt" "$OUT/.phishing-full.new" && mv "$OUT/.phishing-full.new" "$OUT/phishing-full.txt"
for f in "$TMP"/phishing-part-*.txt; do
base=$(basename "$f")
mv "$f" "$OUT/.$base.new" && mv "$OUT/.$base.new" "$OUT/$base"
doneTwo smaller integrations usually follow the same day, and neither needs the feed. The single-lookup endpoint, GET /api/v1/check?domain=<host>&apikey=<key>, is what an analyst hits when a hostname appears in a ticket and they want a verified answer in the sixty seconds before they escalate; the response carries is_phishing, category, dns_status, last_checked, confidence and the current database_size, which is enough to close or promote the ticket. The batch endpoint, POST /api/v1/batch with a JSON body of up to a hundred domains and one credit charged per domain, is what a nightly job uses to sweep the destinations that actually appeared in the firewall's own logs rather than everything in the database.
It travels as a query parameter or in the POST body, and it belongs in the environment of the conversion host or the SOC tooling, never in anything that reaches a browser. If you are building an internal dashboard that displays lookup results, put a small endpoint of your own in front of the API and let the browser talk to that instead.
The binding constraint is almost never the datacentre pair. It is the smallest branch appliance in the fleet.
Every platform documents limits, and there are usually three stacked on top of each other: a maximum number of entries in a single list object, a maximum number of list objects the device will hold, and a global budget of fully-qualified domain name entries shared with every other feature that consumes them. Those numbers differ by hardware model, by software train and sometimes by licence tier, which means the correct first step is not to generate a file but to open the datasheet for the least capable device that will carry this policy. A pair of high-end appliances in the core will swallow anything you give them. The desktop-form-factor box in a twelve-person branch office is where the ceiling lives, and it is the one that decides your file layout for the whole estate.
The mechanics are unremarkable. If the ceiling is fifty thousand entries per object, generate parts of forty-five thousand lines each, define one list object per part, and reference all of them from a single policy rule or address group. Policy complexity does not grow with the number of parts, because the parts are joined at the group level rather than the rule level. Keep the part count stable across days where you can, since adding an object is a configuration change that has to be pushed to every device, whereas changing the contents of an existing object is only a fetch. If the list grows past the last part, the job should create a new part and you should find out that it happened, rather than silently truncating.
This is where the verification standard behind the data does real operational work. Every domain in the database is confirmed to be currently resolving in DNS; entries that go dark are dropped rather than retained. A blocklist that accumulates every hostname ever reported grows without limit and spends object budget on domains that have been unreachable for two years, and on an appliance with a hard entry ceiling that is not a philosophical problem — it is the reason you cannot fit the list. Roughly 390,000 actively-resolving domains is a number you can plan a fleet around. An archive of everything ever observed is not.
Trim before you compromise on refresh. The CSV carries a category column, so the conversion job can filter to the categories that match your exposure and leave the rest — a deliberate, documented reduction is far better than an accidental one. The alternative worth considering first, though, is moving enforcement to a layer with no object ceiling at all: a local resolver holding the full list, with the firewall handling only what the resolver cannot see. That pattern is covered on the DNS filtering page and it is frequently the right answer for an estate whose branch appliances are all small.
On most platforms a list refresh is not free: the object is re-parsed and, depending on the vendor, a policy commit or a dataplane update follows. On a busy device that is a measurable event, which is the practical argument against a five-minute polling interval on a file that changes once a day. Measure the reload on one device during a maintenance window, then set the fleet interval with that number in hand rather than accepting the default.
The two consumption models have different economics, and most firewall teams end up running both for different jobs.
The feed is the model that fits a firewall. It is a separate subscription that provides the complete database as CSV plus the daily changelog; you pull it once a day, and enforcement afterwards costs nothing per request because the matching happens on your own appliance. That is the correct shape for a control that has to answer millions of times a day at line rate: there is no per-connection call to make, no external dependency in the data path, and no upstream service whose availability quietly becomes your availability. If the link to your staging host fails, the appliance keeps enforcing yesterday's list, which is precisely the degradation behaviour you want.
One credit is one API lookup, lookups reset each month, and there is no subscription or per-seat licence on the API itself. That suits the jobs where you have a specific set of hostnames and a specific question: sweeping the distinct destinations from yesterday's proxy logs, enriching an alert before it reaches a human, checking the domains in a report a user forwarded, or validating a set of look-alikes for your own brand before you file takedowns. Those are hundreds or thousands of lookups a day, not millions, and the batch endpoint's hundred-domains-per-request limit keeps the request count sensible.
That is because the packages are fixed. A security team checking a couple of hundred hostnames a day from log analysis sits comfortably inside the Professional package at Professional at $249/month for 100,000 lookups, which works out at $0.0025 per lookup. A larger environment enriching every alert plus running a nightly sweep of egress destinations lands in Business at $499/month for 250,000 lookups, or $0.0020. Payment is by PayPal, the credits do not expire for twelve months, and there is no meter running on the appliance itself, because the appliance is enforcing an imported file rather than calling the API.
10,000 credits at $0.0040 per lookup. Enough to script a proof of value against last month's logs before anyone signs anything.
100,000 credits at $0.0025 per lookup. The usual landing spot for a SOC doing daily enrichment alongside the imported feed.
250,000 credits at $0.0020 per lookup. Suits a multi-site estate where every alert and every egress sweep gets a verified answer.
Three matching points, each with a caveat that decides whether a block lands or a connection quietly succeeds.
If the appliance runs a DNS filter profile, or if your clients resolve through a server that consumes the same list, the hostname is visible in cleartext before any connection exists and the block is as cheap as it will ever be. The user gets a resolution failure or a sinkhole address, and nothing reaches the attacker at all.
The Server Name Indication field carries the hostname the client intends to reach, in cleartext, before the encrypted session is established. That is why a firewall can deny a whole HTTPS destination by name without decrypting anything — the single most useful property on this page. You do not need to break TLS to enforce a domain list.
Visible only on cleartext connections, and increasingly a footnote. It still matters for the small amount of plain HTTP left in most environments, and for the initial request in a redirect chain or a captive-portal handshake, where the first hop is frequently still unencrypted.
Full TLS inspection sits on top of all three and buys you the URL path, the request body and the response content. It also costs a trust anchor deployed to every endpoint, a permanent list of exclusions for applications that pin certificates, a real conversation with whoever represents staff privacy in your organisation, and a throughput budget on the appliance that is often the reason the project stalls. It is worth having, for other reasons. It is not required for this one. A domain list is enforced entirely at the name layer, and the sequencing advice is simple: deploy the list against SNI and DNS now, and treat decryption as a separate project with its own justification.
That is the constraint they share. If a phishing kit is hosted on a large file-sharing service, a form builder, or a compromised page inside an otherwise legitimate site, the hostname is not the malicious unit and a domain list has nothing useful to say. Blocking the host would take down something your organisation genuinely uses. That is the honest ceiling on host-level enforcement, and the answer is not to stretch the list into something it is not — it is to accept that this class of campaign belongs to your mail gateway, your browser controls and your endpoint protection layer instead.
They are sold as alternatives and they are not. Reading the rows below is usually enough to end the debate in a design review.
| Question in the design review | URL-filtering categories | Verified phishing domain list | Running both |
|---|---|---|---|
| What question does it answer? | What kind of site is this, by taxonomy | Is this exact hostname confirmed live phishing infrastructure | Both, from two independent sources of truth |
| Three-day-old look-alike sign-in page | Usually uncategorised or newly registered | Denied if it has been observed and verified | Denied by the list, logged with a category |
| Who decides what is on it? | The vendor's classification pipeline, which you cannot inspect | DNS verification that the host resolves, plus confirmation it is hostile | Two standards of proof, recorded separately in the log |
| Shape of a false positive | A whole legitimate category blocked for a business unit | One hostname wrongly denied, correctable locally in minutes | Two places to look when a user complains |
| Cost on the appliance | A licensed engine, minimal object budget | Object entries against the device ceiling, split across list objects | The engine licence plus the object budget |
| Refresh behaviour | Vendor-controlled, opaque, generally continuous | Rebuilt every 24 hours, pulled on your own schedule | Independent cadences, which is the point |
| Works without TLS decryption | Partly — hostname only, no path categorisation | Yes, hostname matching is all it needs | Yes for both, at the name layer |
| Behaviour when the WAN is down | Depends on whether the engine calls out for verdicts | Enforces the last imported file, unchanged | As weak as whichever half needs the network |
| Effort to add to an existing policy | Enable the profile, then argue about the categories for a month | One conversion job, one list object, one deny rule | Both, but the list half lands first |
The row worth arguing about is the false-positive row, because it is where the two controls differ most in the experience they create. When a category engine over-blocks, an entire business function loses access to a class of sites and the fix is a policy exception that has to be justified and re-reviewed. When a discrete list over-blocks, exactly one hostname is affected, the fix is a local allow entry that takes a minute, and nothing else in the policy changes. That difference in blast radius is why teams who are nervous about the category product are frequently comfortable with the list, and why deploying the list first is a low-risk way to demonstrate value before the harder conversation happens.
It decides what you can evidence afterwards. A vendor reputation service updates on a schedule you neither see nor control, which is fine until the day you need to answer “when did this domain become known-bad, and were we protected before the user clicked?” With an imported list you know exactly which generation was on the box at 14:07, because you built it and you kept the previous one. That auditability is worth more in an incident review than a marginally faster update you cannot evidence.
A block that produces no explanation and no log line has done half its job and thrown the other half away.
A multi-site estate does not need thirty download jobs. It needs one, plus a serious answer to what happens when that one fails.
Branch appliances should not each fetch from the internet. Thirty devices pulling the same multi-megabyte file across thirty different links is wasteful, it requires thirty egress exceptions that somebody will eventually tidy away without knowing what they were for, and it creates thirty independent opportunities for a certificate or proxy problem to leave one site silently stale. One internal staging host, reachable from the management network, is the correct topology: it performs the conversion, it holds the previous generation for rollback, and it is the single object you monitor.
Where it exists — a management centre distributing Security Intelligence to its sensors, or a central manager pushing objects to managed firewalls — use it, because it collapses the version-skew problem entirely. The alternative, letting each device pull independently, is perfectly workable but means you need some way to answer the question “which sites are running which generation of the list?” That answer usually lives in a report you have to build yourself; with central distribution it is a property of the platform.
This is the part teams skip. When the staging host is unreachable, does the appliance keep enforcing the last list it successfully fetched, or does it empty the object? Both behaviours exist in the wild, and the difference is the difference between a degraded day and an unprotected one. Test it deliberately in a maintenance window by denying the appliance access to the staging host and confirming that a known-bad hostname is still refused. Write the result down; it is the first question anybody asks during a real outage.
Alert on the age of the file on the staging host and, separately, on the generation of the list as the appliance itself reports it. Those two can diverge — a perfectly fresh file that nobody is fetching is the classic silent failure — and only the second tells you whether the policy on the box reflects today's data. A single alert on “list older than forty-eight hours” catches essentially every realistic way this deployment decays.
Worth writing into the design document, because every one of these will be raised eventually and it is better raised by you.
A hostname earns its place by being observed in the wild, confirmed to resolve, and identified as credential-harvesting infrastructure. Nothing in that process produces an opinion about a domain nobody has encountered yet, so an absent hostname means only that it is absent — infrastructure stood up at nine and fired at two has not been through the pipeline in time. Every inventory of known-bad indicators behaves this way, and the useful framing on a perimeter device is that you are raising a floor: the destinations already proven hostile are unreachable from your network, and everything else remains the job of controls that judge behaviour rather than names.
Phishing hosted on shared infrastructure your organisation legitimately uses is out of scope by construction, because the only lever available at this layer would take the legitimate use down with the malicious one. Nor does the list see inside an encrypted session: without decryption you are matching names, so a page served from an already-permitted host is invisible to this control regardless of what it contains.
There is a routing gap as well. A device using DNS-over-HTTPS to a public resolver bypasses your DNS filter profile, and a laptop on a home network or a phone tether is not behind your appliance at all — which in a hybrid-work estate is a smaller share of the working day than it used to be. That is the argument for putting the same data in more than one place, the resolver and the mail path and the endpoint, rather than treating the perimeter as complete.
The block page, the report path, the ticket, the follow-up on who else received the same message: those are what turn a prevented click into a closed campaign. Teams who pair the import with the analysis workflow described on our threat intelligence page consistently get more out of it than teams who import the file once and never look at the logs again.
Short answers to the objections that decide whether this ships this quarter or next year.
Matching is not the concern. Domain lists are compiled into a hash or trie structure on the device, so lookup cost is effectively constant whether the object holds five thousand entries or fifty thousand, and it does not scale with the number of concurrent sessions. The steady-state impact on forwarding performance is not something you will be able to measure.
No. The hostname a client intends to reach appears in the TLS ClientHello's Server Name Indication field in cleartext before the encrypted session exists, so a firewall can deny the connection by name without ever holding a key. That is the whole reason a domain list remains a practical control on modern traffic.
Split the list into parts that each fit inside one object and reference them from a single group, which is what the conversion script above does with split -l. Policy complexity does not increase; you gain one object per part and nothing else. Size the parts with headroom so that a month of growth does not require a same-day change on every device in the estate.
Locally, and in about a minute. Keep a small allow-list object defined ahead of the deny rule in the policy, add the hostname to it, and the block stops immediately without waiting for anything upstream. Because the enforcement unit is a single hostname rather than a category, the blast radius of a wrong entry is one destination rather than a business function.
GET /api/v1/check tells you whether the domain is still flagged, and entries that no longer appear in the database can be removed from the exception list entirely.On most platforms the appliance keeps enforcing the last list it successfully fetched, which is the behaviour you want — a stale list is far better than an empty one. But this varies by vendor and by software version, so verify rather than assume: deny the appliance access to the staging host during a maintenance window and confirm that a known-bad hostname is still refused.
Log-only for one to two weeks on a single device, then deny. Not because the data needs proving, but because two weeks of matches from your own network is the artefact that makes the rest of the rollout uncontroversial. It gives you the real hit rate in your environment, surfaces any overlap with something the organisation genuinely uses, and produces a number that came from your traffic rather than from a vendor slide.
Yes, for a specific and useful job: checking the outbound links your own application renders. If your platform hosts user-submitted content — profiles, listings, forum posts, support tickets — the hostnames in those links can be checked server-side before the content is stored or served, so your application never becomes the delivery vehicle for somebody else's campaign.
POST /api/v1/batch from your backend, because you are checking a handful of specific hostnames rather than filtering all traffic.It is an independent source with a different inclusion standard, and the independence is the point. A vendor's built-in reputation service and a separately maintained verified list will not contain the same domains on the same day, and the union of the two is strictly better coverage than either alone. Adding an external list does not disable anything you already run.
One conversion job, one list object, one deny rule high in the policy, and a fortnight of log-only data to bring to the review. The daily CSV feed is built for exactly this import, and monthly plans cover the lookups your analysts make around it.
