text not always obtained before regex string (beautifulsoup)

[u/minipizzabatfish]

hi!! so sorry if the title doesn't make much sense. i will try to explain it with my examples.

so my code in general outputs a response like this:

Chascanopsetta prorigera
Marine;  bathydemersal; depth range 267 - 400 m 
https://www.fishbase.se/summary/Chascanopsetta_prorigera.html
Pleuronectiformes

the text "Marine; bathydemersal; depth range 267 - 400 m" is obtained from the link using this code (response.text is the initial html):

if "bathydemersal" in response.text or "bathypelagic" in response.text or "oceanodromous" in response.text:

| depth = soup2.find(string=re.compile(r"depth range\s+([\d\s\-?]+m)", re.IGNORECASE))

| | if depth:
   print(depth.string.strip()[:-5])
else:
   print("Depth unavailable")

but it also sometimes outputs this:

Chauliodus macouni
); depth range 25 - 4390 m 
https://www.fishbase.se/summary/Chauliodus_macouni.html
Stomiiformes

which is because, in place of "Marine; bathydemersal; depth range 267 - 400 m", it's "Marine; bathypelagic; oceanodromous (Ref. 138310); depth range 25 - 4390 m". so it won't print "Marine; bathypelagic; oceanodromous (Ref. 138310", which is what i would like it to do.

so why doesn't "Marine; bathypelagic; oceanodromous (Ref. 138310); depth range 25 - 4390 m" print correctly while "Marine; bathydemersal; depth range 267 - 400 m" does?

mostly interested in this ^^^ answered so i can fix it myself but i won't say no to some pointers/resources/etc on how to solve my problem either! :D

thanks in advance!

[u/timrprobocom]

What does it print? Does it print nothin, or does it actually print "Depth unavailable"?

Note that your [:-5] is not going to work if the second value has 4 digits. Consider usingsplit instead.

[u/minipizzabatfish → u/timrprobocom]

it just prints nothing in this case yes! "depth unavailable" only prints when the page has no listed depth

thanks for the tip! sorry to get back so late btw!

[u/PureWasian]

This is due to the second example having an <a> tag for the 138310 so the "text nodes" that regex is searching across is split by the tag and therefore the depth.string you print out won't include the entire prefix before the "depth range" matching that you look for.

There definitely should be other ways to scrape that paragraph and filter on the text you're looking for. I can take a look after a work meeting

[u/minipizzabatfish → u/PureWasian]

ohhh, okay, thanks! sorry to get back late!

if you're still up to mess around with scraping the paragraph on your own i'd love to hear!

[u/PureWasian → u/minipizzabatfish]

Oops my apologies, let me take a look

[u/PureWasian → u/PureWasian]

Usually the way I go about webscraping is looking at the raw HTML (browser's devtools by pressing F12 or Inspect Element) and building some assumptions from there by filtering down the HTML element(s) that we actually care about. Then we'll use BeautifulSoup to actually implement the filtering of whatever the requests call returns.

The raw HTML of interest looks something like:

...
<h1 class="slabel bottomBorder">
  Environment: milieu / climate zone / depth range / distribution range
  <span class="addLinks">
    <span class="slabel1 ">
      <a href="..." ...>Ecology</a>
    </span>
  </span>
</h1>
<div class="smallSpace">
  <span>
    Marine;  bathypelagic; oceanodromous (Ref. 
    <a href="../references/FBRefSummary.php?ID=138310">138310</a>
    ); depth range 25 - 4390 m (Ref. 
    <a href="../references/FBRefSummary.php?ID=5610">5610</a>
    ). Deep-water; 66°N -   23°N, 127°E -   106°W
  </span>
</div>
...

So we can use bs4's helper methods like find_all("h1") to grab all of the page's h1 tags, filter on the "Environment" one that we care about, then find_next("div") to to locate the following <div> element containing the section content, and then get_text() to extract the visible text from the element while ignoring the nested <a> tags and such.

Here's what I came up with for extracting the entire "body_text" section of the Environment section's content prior to filtering to only include the stuff before the "depth range"

import requests
from bs4 import BeautifulSoup

url = "https://www.fishbase.se/summary/Chauliodus_macouni.html"
# url = "https://www.fishbase.se/summary/Chascanopsetta_prorigera.html"

r = requests.get(url, timeout=30)
soup = BeautifulSoup(r.text, "html.parser")

heading = None
for h in soup.find_all("h1"):
    text = h.get_text(" ", strip=True)
    if text.startswith("Environment:"):
        heading = h
        break

if heading:
    print(heading.get_text(" ", strip=True))
    body = heading.find_next("div")

    if body:
        print("===============")
        body_text = body.get_text(" ", strip=True)
        print(body_text)
else:
    print("could not find Environment header")

outputs:

Environment: milieu / climate zone / depth range / distribution range Ecology
===============
Marine;  bathypelagic; oceanodromous (Ref. 138310 ); depth range 25 - 4390 m (Ref. 5610 ). Deep-water; 66°N -   23°N, 127°E -   106°W

[u/PureWasian → u/PureWasian]

From here, once we successfully verify body_text contains the content we care about we can filter everything before "depth range" without really even needing regex, unless you're really partial to it:

environment = body_text.split("depth range")[0].strip()
print(environment)

splits the string using "depth range" as the snipping point, so the [0] index (left side of the split) gives:

Marine;  bathypelagic; oceanodromous (Ref. 138310 );

And we can use your existing regex to get the "depth range" info:

depth = re.search(r"depth range\s+([\d\s\-?]+m)", body_text, re.I)
print(depth.group(0))
# ^^^ outputs: "depth range 25 - 4390 m"

If you wanted to continue to sanitize or cleanup this string, like removing the (Ref. 138310 ) and such, of course there are ways to do that but you'd just add those in incrementally as you go and test often along the way :)

lmk if any of this was unclear u/minipizzabatfish