Releases: crwlrsoft/crawler
Releases · crwlrsoft/crawler
v1.8.0
Added
- New methods
Step::keep()andStep::keepAs(), as well asStep::keepFromInput()andStep::keepInputAs(), as alternatives toStep::addToResult()(orStep::addLaterToResult()). Thekeep()method can be called without any argument, to keep all from the output data. It can be called with a string, to keep a certain key or with an array to keep a list of keys. If the step yields scalar value outputs (not an associative array or object with keys) you need to use thekeepAs()method with the key you want the output value to have in the kept data. The methodskeepFromInput()andkeepInputAs()work the same, but uses the input (not the output) that the step receives. Most likely only needed with a first step, to keep data from initial inputs (or in a sub crawler, see below). Kept properties can also be accessed with theStep::useInputKey()method, so you can easily reuse properties from multiple steps ago as input. - New method
Step::outputType()with default implementation returningStepOutputType::Mixed. Please consider implementing this method yourself in all your custom steps, because it is going to be required in v2 of the library. It allows detecting (potential) problems in crawling procedures immediately when starting a run instead of failing after already running a while. - New method
Step::subCrawlerFor(), allowing to fill output properties from an actual full child crawling procedure. As the first argument, you give it a key from the step's output, that the child crawler uses as input(s). As the second argument you need to provide aClosurethat receives a clone of the currentCrawlerwithout steps and with initial inputs, set from the current output. In theClosureyou then define the crawling procedure by adding steps as you're used to do it, and return it. This allows to achieve nested output data, scraped from different (sub-)pages, more flexible and less complicated as with the usual linear crawling procedure andStep::addToResult().
Deprecated
- The
Step::addToResult(),Step::addLaterToResult()andStep::keepInputData()methods. Instead, please use the new keep methods. This can cause some migration work for v2, because especially the add to result methods are a pretty central functionality, but the new "keep" methodology (plus the new sub crawler feature) will make a lot of things easier, less complex and the library will most likely work more efficiently in v2.
Fixed
- When a cache file was generated with compression, and you're trying to read it with a
FileCacheinstance without compression enabled, it also works. When unserializing the file content fails it tries decoding the string first before unserializing it.
v1.7.2
v1.7.1
v1.7.0
Added
- Allow getting the whole decoded JSON as array with the new
Json::all()and also allow to get the whole decoded JSON, when usingJson::get(), inside a mapping using either empty string or*as target. Example:Json::get(['all' => '*']).*only works, when there is no key*in the decoded data.
Fixed
- Make it work with responses loaded by a headless browser. If decoding the input string fails, it now checks if it could be HTML. If that's the case, it extracts the text content of the
<body>and tries to decode this instead.
v1.6.2
v1.6.1
v1.6.0
Added
- Enable dot notation in
Step::addToResult(), so you can get data from nested output, like:$step->addToResult(['url' => 'response.url', 'status' => 'response.status', 'foo' => 'bar']). - When a step adds output properties to the result, and the output contains objects, it tries to serialize those objects to arrays, by calling
__serialize(). If you want an object to be serialized differently for that purpose, you can define atoArrayForAddToResult()method in that class. When that method exists, it's preferred to the__serialize()method. - Implemented above-mentioned
toArrayForAddToResult()method in theRespondedRequestclass, so on every step that somehow yields aRespondedRequestobject, you can use the keysurl,uri,status,headersandbodywith theaddToResult()method. Previously this only worked forHttpsteps, because it defines output key aliases (HttpBase::outputKeyAliases()). Now, in combination with the ability to use dot notation when adding data to the result, if your custom step returns nested output like['response' => RespondedRequest, 'foo' => 'bar'], you can add response data to the result like this$step->addToResult(['url' => 'response.url', 'body' => 'response.body']).
Fixed
- Improvement regarding the timing when a store (
Storeclass instance) is called by the crawler with a final crawling result. When a crawling step initiates a crawling result (so,addToResult()was called on the step instance), the crawler has to wait for all child outputs (resulting from one step-input) until it calls the store, because the child outputs can all add data to the same final result object. But previously this was not only the case for all child outputs starting from a step whereaddToResult()was called, but all children of one initial crawler input. So with this change, in a lot of cases, the store will earlier be called with finishedResultobjects and memory usage will be lowered.
v1.5.3
Fixed
- Merge
HttpBaseLoaderback toHttpLoader. It's probably not a good idea to have multiple loaders. At least not multiple loaders just for HTTP. It should be enough to publicly expose theHeadlessBrowserLoaderHelperviaHttpLoader::browserHelper()for the extension steps. But keep theHttpBasestep, to share the general HTTP functionality implemented there.
v1.5.2
v1.5.1
Fixed
- For being more flexible to build a separate headless browser loader (in an extension package) extract the most basic HTTP loader functionality to a new
HttpBaseLoaderand important functionality for the headless browser loader to a newHeadlessBrowserLoaderHelper. Further, also share functionality from theHttpsteps via a new abstractHttpBasestep. It's considered a fix, because there's no new functionality, just refactoring existing code for better extendability.