Important issue with missed_count

[urinella]

Yesterday I did a -cd because of new spawnpoints were added. Today I looked into the DB and noticed that a lot of spawnpoints are not scanned again ~ 1 hour after starting the initial scan. The missed_count was up to 60 per spawnpoint but they still exist in the game.

Here is a picture. I started the init scan at 09:00 and last_scanned was already 1 hour later.

I think after > 5 missed_count the spawnpoint is skipped.

I also noticed that a missed_count is added in the following situation:

• 2 spawnpoints are really close to each other.
• Spawnpoint A spawns at xx:10. Spawnpoint B spawns at xx:15.
• If spawnpoint A is scanned spanwpoint B gets a missed_count.

Edit:
Here is another picture right after setting missed_count to 0 and scanner restarted. As you can see valid spawns get missed_count too: Time: 07:32:34

And this is after ~ 20 minutes: Legit spawns scanned minutes ago have a count of 8... Time: 07:53:00

This is a huge bug inside the speed-scanner.

Expected Behavior

RM scans all spawnpoints.

Current Behavior

RM is skipping a lot of valid spawnpoints. (See introduction text)

Possible Solution

• Remove missed_count from speed scan
• Still scan spawnpoints with missed_count > 5
• Think about another way to skip non-existing spawns.

[dongemus]

I believe what is happening is Niantic removed those new spawn points ... the scanner is trying to find those spawn timers (TTH) for spawn points that don't exist anymore and that is causing the problem.
I think Niantic might have added a Christmas Event spawn points early on accident

[urinella]

No, like I wrote: These Spawnpoints are still in the game.
New spawns came at 22:30 UTC and were remove at 21:00 UTC the next day. I started the initial scan at 9:00 UTC and at 10:00 UTC they are already out of scan.
And also old spawns are not scanned.
So it's a problem with speed scan and not Niantic.

[crhbetz]

There definitely is something wrong. I did see what @dongemus said - there were new spawn points that seemingly disappeared again after roughly 24 hours. I'm seeing the same issue as @urinella since after this whole mystery spawn point event, though.

I did a -cd at about 2017-12-20 15:00 UTC (yesterday, 16 hours ago as of the time of this comment), with completely fresh accounts because I thought my database was messed up from those spawn points, getting a lot of missed_count in the db.

However the -cd did not help one bit with the missed_count: right now, I'm at a total of 8187 spawn points in my database. Of those, 974 have a missed_count > 0 (that's 12%, 16 hours into a completely fresh database), 525 are at missed_count = 1 and 163 got missed_count > 5.

There is something strange going on because I've never seen any missed_count ever, outside of event spawns shutting down or blinded accounts. I didn't check anything against real-game sightings or watch the issue any further so I can't comment on @urinella 's further observations.

[Tomakava]

It is a bug caused by #2339 jittered location update. By jittering location spawnpoints are randomly assigned to wrong scannedlocation.
As a workaround you should enable "no-jitter" and clean scanspawnpoint table.

[crhbetz]

I did another -cd setting the POGOMAP_NO_JITTER env variable following @Tomakava 's recommendation and about two hours in, I'm not seeing any issues like I did before. On the last run I already had a 3-digit number of spawn points with missed_count > 0 after 2 hours, so I'll say my issue indeed was caused by the jitter update.

[friscoMad]

@Tomakava Can you elaborate on that?
I am unable to find why that PR has something to do as spawnpoints are asigned to steps during the initial startup and when found as new using scanned_location that is the step location not the jittered position.
But it is true that with jittering enabled (even previously to #2339) if you find a spawnpoint it can be missassigned if you jittered so far from the step_location to see a spawnpoint from another spawnpoint, equally if a spawnpoint is in the border of a step the jitter can make it to be just far away so you can not see it during a regular scan and thus triggering a miss.
In general during initial scan jitter should be disabled to avoid issues with TTH findings and spawnpoint initial alocation.
Anyway jitter is disabled by default now by #2406

[urinella]

@Tomakava This did not solve the problem. I did a rescan and I'm on 80+ missed_count after a few minutes. This bug is much deeper in RM and speed scan.
Also if the accounts get blind before the temp ban they add missed_count to the spawnpoints. This is a bug introduced with the decision to merge the speed scan one year ago.

Most usual users don't notice this bug because they have no clue about the DB and data.

[Tomakava]

In my case just after updating to this PR I got huge number of "has no Pokemon" errors. After disabling jittering once again everything is running smooth. Just a couple of missed_count from 14K spawnpoints.

Now I am doing small scale testing with 1 worker st 3 hex scan and I definitely can replicate this problem. After scanning for about half an hour I start to see spawnpoints assigned more than 70 meters form step location. And with this PR reverted it never happened for several hours.

And I am now almost certain that problem is in this line: api.set_position(*scan_coords)
I think jittering was never working before this PR.

[friscoMad]

@Tomakava that makes sense, the jittered location was only used to get the cell ids to request and to send the inside the gmo message but not for positioning the envelope request so efectively maybe we never had jittering working 😄

[urinella]

I don't know if #2406 fixed this problem.
I'm still getting a lot of missed_count on valid spawnpoints.

Maybe a problem with this (nobody mentioned here before)

I also noticed that a missed_count is added in the following situation:

2 spawnpoints are really close to each other.
Spawnpoint A spawns at xx:10. Spawnpoint B spawns at xx:15.
If spawnpoint A is scanned spanwpoint B gets a missed_count.

[sebastienvercammen]

@urinella Running post-#2406 with jitter enabled or disabled?

[urinella]

@sebastienvercammen Yes with jitter disabled, newest update. Losing more and more spawns every day.

[Tomakava]

@urinella have you cleared database or at least scanspawnpoint table after disabling jittering? Without clearing this table disabling jittering does not fix this problem, because wrong spawnpoint associations are persisted in database.

[urinella]

@Tomakava So I just finished a new scan and cleared all s* tables.
Now it looks a bit better but I have still spawns with missed_counts > 5 after a few hours. (Not minutes any more)

Is it possible that Niantic is permanently removing spawnpoints after stops are deleted?

Right after the inital scan I have spawnpoints where last_scanned was yesterday.
I cannot explain this with something different.

[urinella]

Update:
Yesterday I had missed_count = 6 on > 1000 spawnpoints and last_scanning dates from 2018-01-05, 2018-01-06, 2018-01-07.
Then I added a cronjob to set missed_count = 0 to test if these spawns are really not existing anymore. After 12 hours I can see in the database that now only 43 spawnpoints have a last_scanned < today. All others have 2018-01-08. Maybe this 43 were removed with stops.

So for me missed_count is is not working correct even with diabled jittering.
I updated RM on Thursday before I made the new scan with empty: spawnpoint, scannedlocation, spawnpointdetectiondata and scanspawnpoint tables.

If no one else have this problem I cannot explain whats the error here.

[sebastienvercammen]

@urinella The follow-up in this issue has been a bit slow on Github, but we've been keeping track of help requests and I've looked around for people with the same issue.

Unfortunately, I honestly haven't been able to find anyone. Before the jitter rework, there were tons of people with the issue. With jitter disabled, I can find no one besides you. And we've tried everything: from existing databases to new setups, all get a handful of disabled spawns eventually (missed_count > 5) until that number is fixed at a certain point and no longer increases. Similar to the 43 you mentioned.

After that, no increase until Niantic changes the spawnpoints.

We'll keep this issue open, perhaps more people with the same problem can find the thread and jump in.

[ftrueck]

I had the issue with jitter enabled also. Disabling it helped alot but I don't know if this will affect lifespan of the workers.

[sebastienvercammen]

@ftrueck With the already short lifespan of accounts, we've run some tests and disabling jitter did not extend their lifetime.

If, in the future, API usage is undetectable and it comes down to humanization, then we might need to enable it again.