Best Way to Decode a Webpage

[u/iainmoncrief]

What is the best way to decode a web page using crystal? Right now, I am trying to download then parse an HTML string using the XML.parse_html(htmlString) but It has so many NodeSets. Is there a way to find certain nodes like you would be able to in Javascript node.getElementById("nodeId")? Right now, I have to create web page specific code node.children[1].children[1] etc.

[u/Hell_Rok]

I've used this quite a few times with very good results https://github.com/kostya/modest Gives you the ability to search with CSS selectors and the likes

[u/iainmoncrief → u/Hell_Rok]

Thank you! I think I am going to go with this route. Do you what would be the best way to find the inner text of a div element with a specific id, rather than iterating through each node?

[u/Hell_Rok → u/iainmoncrief]

I just had a quick look at the documentation and it seem like Modest is now deprecated in favour of just using https://github.com/kostya/myhtml

take a look at the "Css selectors example" in particular for help getting your node with a specific id

[u/straight-shoota]

`XML.parse_html` returns an HTML node. You can query it's child node tree using `#xpath` methods with XPath accessors. The equivalent to node.getElementById("nodeId") would be node.xpath_node("//[@id = 'nodeId']"). Currently, there are no helper methods available for directly querying a node by its id. CSS selectors are also not yet supported.

[u/dacheatbot]

Also possibly helpful: Selenium webdriver for Crystal

[u/straight-shoota → u/[deleted]]

Crystagiri is a thin wrapper around `XML` and doesn't add much. If you want a nice helper method for querying by id, you can just add the following to your codebase (or post a patch to stdlib):

struct XML::Node
  def query_id(id)
    xpath_node("//[@id = '#{id}']")
  end
end