![]() ![]() I have been building this thing out for about 4 months now and it is very error prone. I think testing is less of a concern and this is the true issue: It will help experienced people diagnose issues without having to read all the comments.Īfter reading your code and your responses below. You should edit your post to include code. I looked at your code you posted in a comment below code. In the scrapers that I have worked on professionally this usually means including error handling code, enqueueing scraper/parsers for retry, notifying when manual intervention is required, allowing multiple parsers to try to parse, etc. OP should expect that scraper to fail and fail often and prepare for that. If the OP ends up using vcr then I would strongly suggest a strategy that employs frequent re-recording using use_vcr_cassette re_record_interval: 2.days or something similar.Īlso if the problem is due to the whole site breaking when the scraper/parser no longer works then the focus needs to be on making that piece robust under failure. In this case he would want to run the test on live data pretty frequently. If the underlying page changes frequently then repeatedly using the same page to test the parser would be wrong. ![]() This seems like he is going through cycles of it working and then failing because the underlying page changes. This is good for the initial development of a scraper for parsing the page. It best used for repeatable/reliable http interactions. ![]() There is a time and a place for using VCR. If you don't need anything this heavy duty (not unlikely), you can also just get by plugging in your own fixture files. It is also annoying to work with if you forget about it, since stuff will fail that should pass or vise versa if it needs to be re-recorded. Note that it doesn't really deeply examine structured data you are sending by default: a colleague accidentally copy pasted some junk into a huge fixture we had to send up, but VCR didn't notice, passing the test, so the interactivity was broken but we didn't know until it was breaking on production. In that regard it's a tool for regression tests, since you have to get it all figured out and working before recording the interaction for the test. you don't mangle stuff you are sending up. I've had to build apps that are primarily reliant on external API's for functionality, and VCR is (mostly) good for ensuring that you don't end up accidentally breaking your interactivity behavior, e.g. +1 on this VCR is basically a mocking suite that uses fixtures generated from recorded interactions. ![]()
0 Comments
Leave a Reply. |