Is it all about the links? Are there other factors to ranking?
Search engines personalize results. Users are frequently shown information from sources they have viewed before and seem to prefer.
You can see this yourself by opening an Incognito window (CTRL + SHIFT + N or COMMAND + SHIFT + N on a Mac).
Search for a political party or for a news item.. Next flip back to a regular browser window and perform the same search. If the news sources that appeared changed you have found an example of personalized search results.
If you frequently visit one news site over others you should expect that a search engine might favour the results from your preferred news site over others as a topic of interest.
Similarly when someone searches for “screws” or another generic term a search engine might decide to who them a local hardware store, large national chain or photographs and clip art.. the results will be different for each individual because algorithms are constantly changing and evolving.
Links are 100% the best way that search engines have to gauge authority. There are however a number of other metrics that can be used to determine how useful and relevant a page is:
Time on site or Session duration and Bounce rate are both incredibly useful ranking signals that should not be ignored.
Pay attention to both of these on your sites and work to improve your time on site, time on page and lowering your bounce rate.