keep() instead of addToResult() and sub crawlers #142
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
New methods `Step::keep()`, `Step::keepAs()`, `Step::keepFromInput()` and `Step::keepInputAs()` as simpler alternatives for `Step::addToResult()`, `Step::addLaterToResult()` and `Step::keepInputData()` which are all deprecated now. The new keep methods add data to a keep array in IO objects. Not creating a Result object and potentially sharing the same Result object for a lot of child outputs, makes the new keep functionality less complex. No need for something like `addLaterToResult()`. Kept properties can also be used with `useInputKey()` which is pretty handy. Another cool new feature are sub crawlers. Any step can now create a sub crawler to fill a property. Example: you have a page about an author with multiple links to detail pages about his books. You can select those links and let a sub crawler fill the author's `books` property with data from the book detail pages. Further also introduce a new `Step::outputType()` method, that returns if a certain step yields outputs that are associate arrays (or objects), scalar values or potentially both (mixed). This helps reduce potential critical problems during a crawler run by validating before the run and throwing an exception (or log error messages).
Make reading a compressed cache file work, even when useCompression was not called on the `FileCache` instance.
Add trailing commas in multi line function calls.
The new method is `outputType()`. Method `outputKey()` is an existing method.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
New methods
Step::keep(),Step::keepAs(),Step::keepFromInput()andStep::keepInputAs()as simpler alternatives forStep::addToResult(),Step::addLaterToResult()andStep::keepInputData()which are all deprecated now. The new keep methods add data to a keep array in IO objects. Not creating a Result object and potentially sharing the same Result object for a lot of child outputs, makes the new keep functionality less complex. No need for something likeaddLaterToResult(). Kept properties can also be used withuseInputKey()which is pretty handy.Another cool new feature are sub crawlers. Any step can now create a sub crawler to fill a property. Example: you have a page about an author with multiple links to detail pages about his books. You can select those links and let a sub crawler fill the author's
booksproperty with data from the book detail pages.Further also introduce a new
Step::outputType()method, that returns if a certain step yields outputs that are associate arrays (or objects), scalar values or potentially both (mixed). This helps reduce potential critical problems during a crawler run by validating before the run and throwing an exception (or log error messages).