API Core PyType fails intermittently/changing errors

[crwilcox]

#6769

Copying from #6769 and @rchen152

@crwilcox It looks like what's happening is that pytype is somehow finding multiple files named path_template.py (and timeout.py and grpc_helpers.py - the error probably differs because it varies which file pytype discovers first). The other thing I noticed is that the file count in the status message ("Analyzing 53 sources with 0 dependencies") differs from the one I see (28 sources). Any idea why there might be two copies of some source files?

EDIT: if there are two files with the same name in different directories, this may be google/pytype#198.

[crwilcox]

One possible issue is not cleaning up the pytype output folder between runs.

[rchen152]

I tried running the pytype test again, with and without removing pytype_output between runs, and I still can't reproduce the intermittent/changing errors. (There is a consistent pytype error in iam.py.)
The steps I'm taking are:

virtualenv --python=python3.6 .venv3
source .venv3/bin/activate
cd google-cloud-python/api_core/
pip install nox
nox -s pytype

Does this match how you're running pytype?

[tseaver]

@rchen You shouldn't need to create a separate virtualenv: nox does that for you when you run nox -s pytype. I get errors (but not all for the same module) whether or not I remove the pytype_output directory before running it. They all look like:

$ nox -re pytype
nox > Running session pytype
nox > Re-using existing virtualenv at /home/tseaver/projects/agendaless/Google/src/gcp/api_core/.nox/pytype.
nox > pip install --upgrade . grpcio >= 1.8.2 grpcio-gcp >= 0.2.2 pytype >= 2018.9.26
nox > pytype 
Analyzing 53 sources with 0 dependencies
ninja: Entering directory `.../api_core/pytype_output'
ninja: error: build.ninja:20: multiple rules generate .../api_core/pytype_output/pyi/google/api_core/general_helpers.pyi [-w dupbuild=err]

but the module / file varies from run to run.

[tseaver]

I believe the error responsible for the "multiple rules generate" warning is in setup.cfg. We should be limiting inputs to the google/ directory.

[tseaver]

When I edit setup.cfg to include only google/, then I see the following error:

$ nox -re pytype
nox > Running session pytype
nox > Re-using existing virtualenv at .../api_core/.nox/pytype.
nox > pip install --upgrade . grpcio >= 1.8.2 grpcio-gcp >= 0.2.2 pytype >= 2018.9.26
nox > pytype 
Analyzing 26 sources with 0 dependencies
ninja: Entering directory `.../api_core/pytype_output'
[3/26] pytype-single --imports_info /h..._core/google/api_core/path_template.py
FAILED: .../api_core/pytype_output/pyi/google/api_core/iam.pyi 
pytype-single --imports_info .../api_core/pytype_output/imports/google.api_core.iam.imports --module-name google.api_core.iam -V 3.6 -o .../api_core/pytype_output/pyi/google/api_core/iam.pyi --analyze-annotated --nofail --quick .../api_core/google/api_core/iam.py
File ".../api_core/google/api_core/iam.py", line 248, in <module>: No attribute 'register' on Type[MutableMapping] [attribute-error]

For more details, see https://github.com/google/pytype/blob/master/docs/errors.md#attribute-error.
[26/26] pytype-single --imports_info /...gle/api_core/operations_v1/__init__.py
ninja: build stopped: cannot make progress due to previous errors.
nox > Command pytype  failed with exit code 1
nox > Session pytype failed.

I don't understand the error: collections.abc.MutableMapping clearly has a register method: it calls it on the dict type in its own source code.

[tseaver]

OTOH, I think the call to register is spurious: the Policy class already derives from MutableMapping.

[rchen152]

Thanks for fixing this! The MutableMapping error is indeed a pytype bug (google/pytype#197).