0

I'm trying to use the FCC's Electronic Comment Filing System (ECFS) to bulk download filings in individual proceedings. They have an API that will return every filing in a proceeding. It returns a URL for individual documents in the format:

https://www.fcc.gov/ecfs/document/10809709027819/1

However, while this works in the browser, it only downloads a placeholder HTML file saying JavaScript it required when I use wget or curl. I tried examining the page in my browser but couldn't find anything like a source URL for the actual PDF.

Is there a way to use wget or curl to get at the actual PDF?

Improve this question

1 Answer 1

Reset to default
1

I took a look at the request that returns the PDF using Firefox's browser tools, and trimmed down the request headers to get the minimal request that returns a PDF. This works for me:

% curl 'https://www.fcc.gov/ecfs/documents/10809709027819/1' -H 'Referer: https://www.fcc.gov/ecfs/document/10809709027819/1' | file - % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 146k 0 146k 0 0 516k 0 --:--:-- --:--:-- --:--:-- 518k /dev/stdin: PDF document, version 1.7, 9 page(s)

So at least for this PDF, it looks like all that's needed is to set the Referer header to the same URL.

Improve this answer
3
  • 2
    You don't even need the Referer; Notice there's a slight difference between the URL (documents) and the Referer (document). So it works with the plural form (documents instead of document) without the Referer. Commented Aug 14, 2023 at 10:21
  • Huh, I didnt notice that all, which is why I didn't try eliminating that header. Thanks! Commented Aug 14, 2023 at 10:43
  • Is this still working? The method(s) above seem to be getting blocked by the FCC's end... or am i missing some secret sauce. Commented Jul 3 at 17:33

You must log in to answer this question.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.