2

I was wondering if someone could help me out with a web scraping problem.. I am new to both python and web scraping..

I am trying to use the below python program to scrape data from the following webpage,

"https://www.cocorahs.org/ViewData/ListDailyPrecipReports.aspx"

but I keep getting a Runtime error "table#ucReportList_ReportGrid not found" on the raise RuntimeError line (3rd last line).

Now, this program works perfectly fine for a webpage without checkboxes.. However, the webpage I am trying to scrape the data from has some checkboxes and I think that's exactly the part I'm messing up with..

I did a view page source and looked up the html tag names but I feel I am missing something very conceptual.. I am wondering what it might be !!

The two lines after the FL-BV-163 line are the ones I am worried about..

Below is the code. What changes should I make/add to it ? Should I be using some more libraries ? Thank You !!

import requests from bs4 import BeautifulSoup from requests_html import HTMLSession import pandas as pd from io import StringIO from datetime import datetime session = requests.Session() response = session.get('https://www.cocorahs.org/ViewData/ListDailyPrecipReports.aspx') soup = BeautifulSoup(response.content, "html.parser") view_state = soup.find("input", {"name": "__VIEWSTATE", "value": True})["value"] view_state_generator = soup.find("input", {"name": "__VIEWSTATEGENERATOR", "value": True})["value"] event_validation = soup.find("input", {"name": "__EVENTVALIDATION", "value": True})["value"] response = session.post('https://www.cocorahs.org/ViewData/ListDailyPrecipReports.aspx', data={ "__EVENTTARGET": "", "__EVENTARGUMENT": "", "__LASTFOCUS": "", "VAM_Group": "", "__VIEWSTATE": view_state, "VAM_JSE": "1", "__VIEWSTATEGENERATOR": view_state_generator, "__EVENTVALIDATION": event_validation, "obsSwitcher:ddlObsUnits": "usunits", "frmPrecipReportSearch:ucStationTextFieldsFilter:tbTextFieldValue": "FL-BV-163", "frmPrecipReportSearch:ucStationTextFieldsFilter:cblTextFieldsToSearch:0": "checked", "frmPrecipReportSearch:ucStationTextFieldsFilter:cblTextFieldsToSearch:1": "", "frmPrecipReportSearch:ucStateCountyFilter:ddlCountry": "allcountries", "frmPrecipReportSearch:ucDateRangeFilter:dcStartDate:di": "6/13/2025", "frmPrecipReportSearch:ucDateRangeFilter:dcStartDate:hfDate": "2025-06-13", "frmPrecipReportSearch:ucDateRangeFilter:dcEndDate:di": "6/16/2025", "frmPrecipReportSearch:ucDateRangeFilter:dcEndDate:hfDate": "2025-06-16", "frmPrecipReportSearch:ddlPrecipField": "GaugeCatch", "frmPrecipReportSearch:ucPrecipValueFilter:ddlOperator": "LessEqual", "frmPrecipReportSearch:ucPrecipValueFilter:tbPrecipValue:tbPrecip": "0.15", "frmPrecipReportSearch:btnSearch": "Search", }) table = BeautifulSoup(response.content, "html.parser").find("table", id="ucReportList_ReportGrid") if table is None: raise RuntimeError("table#ucReportList_ReportGrid not found") df = pd.read_html(StringIO(str(table)))[0] print(df)
Improve this question
2
  • What data are you trying to scrape? Commented Jun 18 at 8:39
  • @Ramrab: informally speaking the rainfall captured by rain gauge, snowfall etc for a bunch of weather stations for a particular range of dates.. The specific fields which i need are Obs. Date, Station Number and Gauge Catch in. Commented Jun 18 at 18:28

2 Answers 2

Reset to default
4

There's errors in your form data. Correct values for the parameters in question are:

"frmPrecipReportSearch:ucStationTextFieldsFilter:cblTextFieldsToSearch:0": "on", "frmPrecipReportSearch:ucStateCountyFilter:ddlCountry": "0", "frmPrecipReportSearch:ddlPrecipField": "TotalPrecipAmt",

also, then remove: "frmPrecipReportSearch:ucStationTextFieldsFilter:cblTextFieldsToSearch:1": "",

Full code:

import pandas as pd from io import StringIO from datetime import datetime session = requests.Session() response = session.get('https://www.cocorahs.org/ViewData/ListDailyPrecipReports.aspx') soup = BeautifulSoup(response.content, "html.parser") view_state = soup.find("input", {"name": "__VIEWSTATE", "value": True})["value"] view_state_generator = soup.find("input", {"name": "__VIEWSTATEGENERATOR", "value": True})["value"] event_validation = soup.find("input", {"name": "__EVENTVALIDATION", "value": True})["value"] headers = { 'user-agent':'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/137.0.0.0 Safari/537.36'} data={"__EVENTTARGET": "", "__EVENTARGUMENT": "", "__LASTFOCUS": "", "VAM_Group": "", "__VIEWSTATE": view_state, "VAM_JSE": "1", "__VIEWSTATEGENERATOR": view_state_generator, "__EVENTVALIDATION": event_validation, "obsSwitcher:ddlObsUnits": "usunits", "frmPrecipReportSearch:ucStationTextFieldsFilter:tbTextFieldValue": "FL-BV-163", "frmPrecipReportSearch:ucStationTextFieldsFilter:cblTextFieldsToSearch:0": "on", "frmPrecipReportSearch:ucStateCountyFilter:ddlCountry": "0", "frmPrecipReportSearch:ucDateRangeFilter:dcStartDate:di": "6/13/2025", "frmPrecipReportSearch:ucDateRangeFilter:dcStartDate:hfDate": "2025-06-13", "frmPrecipReportSearch:ucDateRangeFilter:dcEndDate:di": "6/16/2025", "frmPrecipReportSearch:ucDateRangeFilter:dcEndDate:hfDate": "2025-06-16", "frmPrecipReportSearch:ddlPrecipField": "TotalPrecipAmt", "frmPrecipReportSearch:ucPrecipValueFilter:ddlOperator": "LessEqual", "frmPrecipReportSearch:ucPrecipValueFilter:tbPrecipValue:tbPrecip": "0.15", "frmPrecipReportSearch:btnSearch": "Search" } response = session.post('https://www.cocorahs.org/ViewData/ListDailyPrecipReports.aspx', headers=headers, data=data) table = BeautifulSoup(response.content, "html.parser").find("table", id="ucReportList_ReportGrid") if table is None: raise RuntimeError("table#ucReportList_ReportGrid not found") df = pd.read_html(StringIO(str(table)))[0]

Output:

print(df.to_string()) 0 1 2 3 4 5 6 7 8 9 10 11 0 Obs Date ▲ Obs Time Station Number Station Name Gauge Catch in. 24hr Snowfall in in SLR Snowpack in in Density Notes State County View Maps 1 6/16/2025 8:00 AM FL-BV-163 Melbourne 4.7 SSE 0.00 0.0 NA NA NA NA NA NaN FL Brevard NaN Active | Static 2 6/14/2025 8:00 AM FL-BV-163 Melbourne 4.7 SSE 0.00 0.0 NA NA NA NA NA NaN FL Brevard NaN Active | Static 3 6/13/2025 8:00 AM FL-BV-163 Melbourne 4.7 SSE 0.00 0.0 NA NA NA NA NA NaN FL Brevard NaN Active | Static

enter image description here

Improve this answer
Sign up to request clarification or add additional context in comments.

5 Comments

The second bs parse could be replaced by df = pd.read_html(StringIO(response.content.decode('utf-8')), flavor='bs4', attrs = {'id': 'ucReportList_ReportGrid'})[0]
@chitown88: Thank you so much !! It works perfectly !! I understand my mistakes, now.
A quick question that I expect to help me with my self-learning.. This is about the "frmPrecipReportSearch:ucStationTextFieldsFilter:cblTextFieldsToSearch:0": "on", line.. How did you know/figure out that there should be an "on" in the end ? The view source page only showed the tag name till "frmPrecip........ToSearch:0" checked="checked" As a self learner, how do you suggest I learn these aspects of coding ? Thank you !!
Use "Inspect" (shft-ctrl-i). You'll see a "Network" tab, under there you can see all the requests made. So try selecting a few different parameters on the site, then click search, you'll the request made in the panel. Find and you'll see the "payload" used in sending the GET.
I see !! Thank you so much !!
0

Form request contains an invalid value

"frmPrecipReportSearch:ucStationTextFieldsFilter:cblTextFieldsToSearch:1": ""

it should be on, off or not sent at all. Sending an empty string makes the site to reply with a different page.

In any case troubleshooting these type of errors starts by inspecting response.content and making sure it contains the expected html response.

Improve this answer

3 Comments

I tried with on, off and removing the two lines. I am still getting the "table.....not found" error.. Thanks though !!
@new_coder check for missing headers too. On web console copy the request as curl and mimic that on python
Thank you !! Turns out the headers were missing as shown by the correct solution !!

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.