> When I need to extract data from HTML I use XPath. But to do so I have to ...

reirob · on July 8, 2011

Thanks for pointing this out. I forgot to precise that I am using XPath on Unix/Cygwin command line or in Shell scripts. Here an example of extracting the Hacker News title;url;points;user data:

   curl -s http://news.ycombinator.com/news | tidy -quiet -asxml -numeric -utf8 -file /dev/null | xmlstarlet sel -N x=http://www.w3.org/1999/xhtml -t -m "//x:tr[x:td[1][@class='title']]" -v "normalize-space(x:td[3][@class='title']/x:a)" -o ";" -v "x:td[3][@class='title']/x:a/@href" -o ";" -v "str:tokenize(following-sibling::x:tr[1]/x:td[2]/x:span[1], ' ')[1]" -o ";" -v "following-sibling::x:tr[1]/x:td[2]/x:a[1]" -n | xmlstarlet unesc

sukuriant · on July 8, 2011

Could you put some newlines in that? For some reason, my browser (IE9) is rendering that construct as a VERY long line and extending the page to match it. (no code-specific scroll-bar for me)

reirob · on July 9, 2011

  curl -s http://news.ycombinator.com/news | \
	tidy -quiet -asxml -numeric -utf8 -file /dev/null | \
	xmlstarlet sel \
		-N x=http://www.w3.org/1999/xhtml \
		-t -m "//x:tr[x:td[1][@class='title']]" \
		-v "normalize-space(x:td[3][@class='title']/x:a)" -o ";" \
		-v "x:td[3][@class='title']/x:a/@href" -o ";" \
		-v "str:tokenize(following-sibling::x:tr[1]/x:td[2]/x:span[1], ' ')[1]" -o ";" \
		-v "following-sibling::x:tr[1]/x:td[2]/x:a[1]" -n | \
	xmlstarlet unesc

sukuriant · on July 9, 2011

Thank you.

yuvipanda · on July 8, 2011

It's also much faster than BeautifulSoup, and uses much less memory. Matters if you're running in a memory/CPU constrained VPS like I was :)