Make test matrix narrower or make it run less frequently

[egeakman]

The current setup runs 56 checks on every push if it's to a branch with an open PR, half of it if there is no PR. This is too much, let's make the test matrix narrower.