4

I'm working on a method that accepts a string (html code) and returns an array that contains all the links contained with in.

I've seen a few options for things like html ability pack but It seems a little more complicated than this project calls for

I'm also interested in using regular expression because i don't have much experience with it in general and i think this would be a good learning opportunity.

My code thus far is

 WebClient client = new WebClient(); string htmlCode = client.DownloadString(p); Regex exp = new Regex(@"http://(www\.)?([^\.]+)\.com", RegexOptions.IgnoreCase); string[] test = exp.Split(htmlCode);

but I'm not getting the results I want because I'm still working on the regular expression

sudo code for what I'm looking for is "

Improve this question
3
  • Though not the answer, this is probably a tool you'll want to use to learn regular expressions: rubular.com Commented Dec 9, 2009 at 16:59
  • Since we're offering useful teach-yourself-regex tools, I'm going to say I'm a big fan of The Regex Coach. It looks to be a bit more feature-full than Rubular (the step-through and tree functions are particularly useful), but obviously requires download and install rather than being browser-based. Commented Dec 9, 2009 at 18:15
  • 1
    Obligatory link: stackoverflow.com/questions/1732348/… Commented Dec 9, 2009 at 18:15

4 Answers 4

Reset to default
3

If you are looking for a fool proof solution regular expressions are not your answers. They are fundamentally limited and cannot be used to reliably parse out links, or other tags for that matter, from an HTML file due to the complexity of the HTML language.

Instead you'll need to use an actual HTML DOM API to parse out links.

Improve this answer
Sign up to request clarification or add additional context in comments.

1 Comment

+1. Specifically, regular expressions and recognize regular languages. HTML is a context-free language, so you need a more powerful parser than a regular expression language to get it 100% right. (en.wikipedia.org/wiki/Chomsky_hierarchy) It might be possible to get a mostly right solution, especially if your regular expression engine has some extensions beyond strict regular grammars. But it's very important to understand the limits of regular expressions. I've seen many security problems caused by not understanding this.
2

Regular Expressions are not the best idea for HTML.

see previous questions:

Rather, you want something that already knows how to parse the DOM; otherwise, you're re-inventing the wheel.

Improve this answer

Comments

2

Other users may tell you "No, Stop! Regular expressions should not mix with HTML! It's like mixing bleach and ammonia!". There is a lot of wisdom in that advice, but it's not the full story.

The truth is that regular expressions work just fine for collecting commonly formatted links. However, a better approach would be to use a dedicated tool for this type of thing, such as the HtmlAgilityPack.

If you use regular expressions, you may match 99.9% of the links, but you may miss on rare unanticipated corner cases or malformed html data.

Here's a function I put together that uses the HtmlAgilityPack to meet your requirements:

 private static IEnumerable<string> DocumentLinks(string sourceHtml) { HtmlDocument sourceDocument = new HtmlDocument(); sourceDocument.LoadHtml(sourceHtml); return (IEnumerable<string>)sourceDocument.DocumentNode .SelectNodes("//a[@href!='#']") .Select(n => n.GetAttributeValue("href","")); }

This function creates a new HtmlAgilityPack.HtmlDocument, loads a string containing HTML into it, and then uses an xpath query "//a[@href!='#']" to select all of the links on the page that do not point to "#". Then I use the LINQ extension Select to convert the HtmlNodeCollection into a list of strings containing the value of the href attribute - where the link is pointing to.

Here's an example use:

 List<string> links = DocumentLinks((new WebClient()) .DownloadString("http://google.com")).ToList(); Debugger.Break();

This should be a lot more effective than regular expressions.

Improve this answer

Comments

0

You could look for anything that is sort-of-like a url for http/https schema. This is not HTML proof, but it will get you things that looks like http URLs, which is what you need, I suspect. You can add more sachems, and domains.
The regex looks for things that look like URL "in" href attributes (not strictly).

class Program { static void Main(string[] args) { const string pattern = @"href=[""'](?<url>(http|https)://[^/]*?\.(com|org|net|gov))(/.*)?[""']"; var regex = new Regex(pattern); var urls = new string[] { "href='http://company.com'", "href=\"https://company.com\"", "href='http://company.org'", "href='http://company.org/'", "href='http://company.org/path'", }; foreach (var url in urls) { Match match = regex.Match(url); if (match.Success) { Console.WriteLine("{0} -> {1}", url, match.Groups["url"].Value); } } } }

output:

href='http://company.com' -> http://company.com
href="https://company.com" -> https://company.com
href='http://company.org' -> http://company.org
href='http://company.org/' -> http://company.org
href='http://company.org/path' -> http://company.org

Improve this answer

Comments

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.