Auto cleanse recordings / temp / cache folder

[Sandy-Garrido]

Hey! Again, thank you for this wonderful repository of pure goodness!

I'm aware that recordings & temp folders can fill up rapidly.
I have had a look in the docs to see if there's a way to either auto clean temp or clear after session close. Is there such way or is there an approach you'd recommend?

I don't want to just periodically clear out the cache as I don't want to remove any required things for the next agent.

tbf, I just saw You can safely discard any databases you do not want to inspect at any future point, or you can archive them to another location. Either approach will not impact future scripts. so I would suspect I'm okay auto cleansing the directory myself.

Would there be any adverse affects if I were to delete the temp whilst a session is in progress? I do not need any stored data really as I'm sending the utilising and then discarding the data straight away.

I'm also spinning up a new agent for each url I scrape... I guess I need to figure out a better approach there but that can come later

[blakebyrnes]

Hi @Sandy-Garrido, thanks for the compliments! :)

You can definitely auto-cleanse the directory yourself. We will likely provide an automated solution for this at some point, but haven't gotten to it yet.

As you suspect, there are a few things you should not delete during runtime:

• Running session databases (ie, 00bebb10-ab82-11eb-a3e0-6721722cb823.db probably use last modified timestamp?)
• sessions.db
• network.db
• *.der

[Sandy-Garrido]

Yeah, I figured! thats great! So I just came up with a solution to delete the .db after agent.close()

const sessionId = await agent.sessionId;
await agent.close();

if (sessionId) {
	clearDbFile(sessionId);
}

export const clearDbFile = (dbName) => {
	try {
		const dbPath = process.env.SESSION_DIRECTORY + "/" + dbName + ".db";
		fs.unlink(dbPath, () => {
		});
	} catch (err) {
		logger.error(`Unable to clear db for session ${dbName}: ${err}`);
	}
};

This seems to have kept my tmp folder at bay!

[blakebyrnes]

Cool! Assuming your env var is self-supplied, that should work.

FWIW, one of the core ideas around the sessions is "when you get blocked" or "a page changes", you can load up the session to see what happened. As long as you've gotten the data you need, probably safe to discard!

[Sandy-Garrido]

Cool! Assuming your env var is self-supplied, that should work.

FWIW, one of the core ideas around the sessions is "when you get blocked" or "a page changes", you can load up the session to see what happened. As long as you've gotten the data you need, probably safe to discard!

That's I good point, I may add some error checking before deletion.