The development of search engines does not stand still. New functions and algorithms are introduced every year. Users have become so accustomed to some that it seems as if they have always been there. One such search feature is search suggestions.

What is this

They are a convenient tool in the Yandex and Google systems, allowing them to simplify and speed up the work of users in finding the necessary information. When entering a word into the search bar, the user sees all possible proposals on this topic. In this case, the possible options change depending on the next word entered in the line.

Google was the first to introduce them. This happened in 2004. After another 4 years, these system capabilities were improved. In 2010, the functionality became even better - now you can see the results loading as each word is entered.

The mechanism, which displays only relevant, that is, recently entered, phrases in the tips, led SEO optimizers to the idea that this can be used to create the semantic core of the site. With this approach, the site’s content will be as relevant as possible to users’ requests.

How search suggestions are generated in Google and Yandex

Based on real queries on the web. The more popular the request, the more likely it is to be included in the list.

Also, the formation is directly related to:

• history of previous requests for this user;
• relevance - the system will most likely display those keywords and phrases that were used recently;
• the language that is chosen as the main one used;
• geo-affiliation - the most popular queries in a particular region or city are shown.

Search engines update suggestions to keep the search functionality up to date. Removal occurs in the following cases:

1. If the hint has lost its relevance.
2. If you have personal information.
3. At the request of the court.
4. In case of obvious.
5. Calls for violence or racial hatred.
6. Related to pornographic topics.

The last 2 options are eliminated immediately, at the first stage of verification.

Many optimizers try to “put” the brand in the tips. Although search engines are actively fighting this phenomenon, sometimes they miss brands in the search results:

• when compiled to encourage users to search for a site, it is possible to provide a large number of queries for a given brand. Which, in turn, becomes the brand’s “pass ticket” to the tips;
• participation in large-scale events also attracts a large number of users looking for information about it. If a brand is sponsoring or participating in an event, this may help it appear in tooltips, but is less likely than the previous method;
• The "In the Tip" service also makes it possible to appear in the search results upon request. This happens by adding a branded request to the service. Thus, the task is given to participants, selected by their geographical location, to search for a site for a given request. This allows us to ensure not only the rapid growth of requests for the brand, but also long-term interest. This situation helps to get into search engines.

How to get a list of search suggestions

Can be obtained in several ways:

1. Using the additional search block in Google “Together with ... often searched for.” To do this, enter a query, and you can see what is being searched for along with it.

This view is not entirely convenient, because, firstly, it shows only about a dozen tips, which is very little for creating the semantic core of the site. Secondly, each phrase has to be typed separately, which takes a lot of time. You cannot download results for several queries at once.

2. Using search engine suggestions parser tools:

This service will tell you what words users are looking for along with the given “keywords”. Additional words are separated from the specified main one by the “+” sign. Alphabetical sorting makes working with data convenient for an SEO optimizer.

• Keyword Tool is a free service. It is based on Google suggestions for different regions and languages ​​of a given search engine. The service also allows you to see semantics from Bing, AppStore and YouTube. As a standard, a word is entered into the search bar and selected by database and language.
• Sloder is a program that allows you to work with several search engines at once, since it works through a proxy server. To get started, you need to download the program, after launching it, enter the search word and select the desired search engine - Google, Mail, Rambler, Yahoo, Nigma. Then you need to click on the “Parse” button and wait for the results to be displayed.
• autodreem

You can find out the latest information about what the design of a successful website should be like.

Where to find search tips in the form of questions and how to collect

These services are convenient in their own way. But there is a service that is most suitable for an SEO optimizer to create a semantic core for a specific site. This service is called Prodvigator.

Its peculiarity is that it makes it possible to receive search engine tips in the form of questions. These questions are real user requests. By answering these questions, you can attract a large number of visitors to the resource. Using the service is quite simple:

Now, using these words and phrases from the online collector, you can promote an information resource. They will perfectly complement the existing semantic core of the site.

• Going to the “Questions Only” tab will help you see a list of phrases that consist only of questions. They can be used to write an article that is relevant to user queries.
• The “batch export” function will allow you to view search suggestions for several key phrases. The service allows you to enter up to 200 such phrases and receive search tips for each in the form of a report. To do this, key phrases are entered into the required field, the results are filtered by toponyms and only interrogative options for clues are selected.

Using search tips to create or expand an existing semantic core allows you to get a list of relevant and relevant phrases that will definitely lead to the site. After all, these are exactly the queries that real users use to find goods and/or services.

But it is also worth taking into account the fact that many users try to remove hints from the search engine. And it’s quite rare to turn them on again.

Sincerely, Nastya Chekhova

SEO Specialist Assistant at SiteClinic.ru

Suggestions in the search bar are an effective method of promotion, but not everyone uses them for good purposes. I recently came across an article about how Google and Facebook manipulated voters using messaging during the 2016 US presidential election. I decided to study this topic in more detail.

In the spring of 2018, American experts published the results of a large-scale study of manipulations with search tips. The authors of the project have developed the first working method for identifying “falsified” tips on Google. In this article I will talk in detail about this method, as well as how to identify fake hints in Yandex.

Search tips or sadgests (from the English suggestion - “suggestion”) are one of the most relevant topics in Internet marketing. Web analysts suggest that this tool will become one of the ways to promote and redirect traffic.

But what if sadgests are already being used for promotion? After all, when entering a search query, we often click on the first hint without thinking (especially when using a mobile device).

How to get into search suggestions

Hints are pop-up options for queries in a search engine. They duplicate the beginning of the text entered by the user. Suggestions help the user accurately formulate a request and select the most relevant results.

In the Google search engine, unlike Yandex, search suggestions have “geography”, but no “targeting”. For example, at the request of a user who is located in Belgorod, he will be shown a georeference. However, there may also be inquiries regarding Kyiv and Kharkov. And it is not a fact that Belgorod or even Russia will be mentioned. Yandex provides a more accurate location link, and the number of messages that appear is much greater. Both systems use "user memory" and will prioritize queries that have already been entered on the device.

Search engines do not officially use hints for commercial purposes, but it is still possible to manipulate traffic and influence the subconscious with their help.

Autofill as a way to boost search suggestions

Once suggestions appeared in the search bar, enterprising SEO specialists began using them to increase traffic to their sites. A large number of fake requests are created mentioning a certain term (name of an organization, product, etc.) in order to direct visitors to the “right” sites and imitate demand:

This type of promotion not only reduces the quality of search results, but also negatively affects the reputation of the search engine. For many commercial HF queries, such “impurities” appear in the list of suggestions, and all search engines, including Google, Yandex, Bing, Yahoo, are victims of this attack.

According to Google, 60% of today's search queries come from mobile devices. Here, form factors make it difficult to enter a query, so tablet and phone users, in most cases, rely on ready-made autofill options. This is how mobile users are manipulated by hints.

How to spot fake suggestions on Google

Not long ago, experts from three American universities conducted a large-scale experiment with manipulation using autocomplete. The study was based on automatic detection of manipulative offers without access to query logs. Scientists have developed the Sacabuche (Search AutoComplete Abuse Checking) method to solve this problem.

According to this approach, manipulative cues can be identified by semantic inconsistency between trigger phrases (keywords within a trigger) and their corresponding target phrases (keywords within a sentence). These tips contain general language such as “reviews”, “companies”, “listing”, “services”. Manipulative messages are more specific (because they are used to promote a product). For example, after entering the trigger “online backup free download”, the researchers discovered that the unfamiliar word “strongvault” appeared in its sentence:

It turned out that this was malicious software that was included in the autocomplete list due to manipulation.

How does Sacabuche work?

Figure 5 shows the architecture of Sacabuche, including the Prediction Searcher (PI), Search Term Analyzer (STA), and Search Results Analyzer (SRA). The UI is designed to detect a large number of automated offers. Specifically, it iteratively queries search engines with a depth limit of 3, starting with a set of triggers as input. This allows you to get more autocompletes. These sentences are further analyzed by APT, which considers a set of semantic features to identify suspicious terms. Such terms are then queried in ARP search engines and their results are validated against content characteristics to capture guided predictions.

The Sacabuche method uses a two-step approach that is semantics-based and minimizes its impact on performance. It uses natural language processing to analyze large numbers of trigger and sentence combinations without querying search engines. This way, the vast majority of real clues are filtered out. And only a small set of suspicious language is launched against search engines to obtain query results to detect manipulation. A manipulator can create a large number of queries, but it is much more difficult to create many relevant results indexed by search engines, so this feature helps distinguish problematic offers from legitimate ones.

The effectiveness of the approach is confirmed by an accuracy of more than 96.23% and a recall of 95.63%, and its scalability allowed the study to be conducted on 114 million sentences.

To automatically identify manipulators, the researchers used a set of NLP (natural language processing) technologies:

• Vector representation of words. It is a general name for various approaches to language modeling based on natural speech processing. These methods are aimed at matching words from a certain dictionary of vectors. The theoretical basis for vector representations is distributive semantics. The vector representation is intended to ensure that synonyms are assigned similar vectors and antonyms are mapped to dissimilar vectors. The study used the popular word embedding tool Word2Vec, which runs on an artificial neural network to build the model and generate vectors. This tool compared the semantic meanings of different words and measured the cosine distance between vectors. For example, the embedding technology automatically identifies words that are semantically close to “casino,” such as “gambling” (cosine distance 0.35), “vegas” (0.46), and “blackjack” (0.48).
• Dependency analysis. Syntax analysis is an NLP technique for describing the grammatical relationships between words in a sentence. Such relations include direct object, determiner, compound noun modifier, etc. A modern dependency analyzer such as the Stanford parser achieves 92.2% accuracy in detecting grammatical relations in a sentence.
• Lemmatization. A natural language document always contains words in different forms, due to temporary abbreviations and grammatical needs. For example, “organize,” “organizes,” and “organization.” In addition, there are words of derivation with similar meanings, such as “slow” and “slowness.” Therefore, you need to find out the original form of each word and then associate them with expressions in different forms. This can be done using lemmatization techniques that reduce inflectional forms, remove inflectional endings, and return the base or dictionary form. A common lemmatization algorithm is morphological analysis, which allows you to figure out the lemma for each word. The modern WordNetLemmatizer algorithm allows you to achieve 95% accuracy.
• The opposite model. Manipulators create a large number of requests in various sources to distribute illegal, unwanted or unrelated content. This makes the IP ID-based discovery approach less effective. However, on the other hand, it is assumed that it is difficult for such manipulators to create a large amount of web content, distribute it on authoritative websites and get it indexed by search engines. Such measures, of course, require more costs and resources than fake requests.

It is obvious that manipulative cues are “given away” by semantic inconsistency. This is because the trigger and its clause are less coupled when autocomplete is manipulated. This is because they are promoting an obscure product that is less relevant to the relevant trigger. For example, “play online bingo games online at moonbingo. com" and "free sites for bingo players" are suggestions for the trigger "bingo sites". The first one being manipulated is more specific (promoting moonbingo.com, a bingo site) and therefore less trigger-like.

In addition to semantic inconsistency, search results for fake clues were found to be inconsistent with their corresponding triggers, while "good" ones were consistent with them. This is due to the fact that the manipulative offer affects the search system’s prioritization: the promoted content does not make it more visible in the search results.

Figure 6 shows the inconsistency between the search results for fake and regular offers. For the "benign" search results for the "Norton online backup free" offer, they were similar to the searches for "free online backup download". At the same time, in the top 20 search results for this trigger there is not a single offer that was offered to us by the hint “strongvault online backup for free.”

According to the results of the study, it turned out that 3 thousand compromised sites in Google's TOP 10 were actually associated with fake search suggestions.

How to detect fake tips in Yandex?

The Google and Yandex hint systems have a concept called “freshness”. For example, if a query becomes popular in a short time, it has a chance to be included in the tips. But it may also disappear after a while due to a decline in interest.

Yandex updates hints at least once a day. Requests that are no longer relevant are deleted. This PS monitors the growth of their number. Therefore, current queries that are of interest to a large number of users are included in Yandex’s “quick” suggestions. They are updated every half hour. This selection occurs on the basis of a sharp increase in interest in events (for example, the latest news, and new publications and social networks).

How do you understand that a request is being “boosted”? Let me give an example of obvious manipulation with suggestions: a case where a promoted site blocked all Yandex search suggestions for the key “education”:

So, you can determine whether a request is being cheated by indirect factors. For example, if among the information prompts for a commercial request you come across a prompt containing the name of an unknown company.

In the Yandex search engine, you can look at Wordstat - the query history. She can tell you a lot. If the request frequency in one month increases from 100 impressions to 10,000, this should alert you. Such figures refer to the first example of obvious cheating.

Analysis of search logs and sources for creating proposals can also detect such manipulations. However, this approach can only be performed by a search provider such as Yandex or Google. Even if search logs are taken into account, careful analysis of huge amounts of data is not trivial.

conclusions

The topic of manipulation with search tips is very relevant. The number of companies and services that offer promotion services through sales promotions is growing rapidly. However, this method reduces the quality of search results and negatively affects the reputation of the search engine. Due to the fact that the number of tips reaches more than a hundred million, and they are constantly updated, such manipulations are difficult to track. This can be used by ill-wishers for phishing, spreading malware, or selling traffic through an affiliate program. In my opinion, these are the main problems of this method.

You can read more about the study.

Any search engine, be it Yandex, Google, Bing or their lesser-known and popular analogues, displays hints when you enter a query into a line. These are their default settings, and this significantly simplifies and speeds up the search process. In the presented list of options, you can quickly find the one you need so as not to enter it completely manually. However, some users are not satisfied with such a convenient search engine operation, and they want to disable hints. We'll tell you how to do this in the Yandex system.

Removing hints in Yandex

There is only one option to disable hints in the Yandex search bar. The steps required to deactivate this useful feature are performed on the search engine's home page, so you can use any web browser. In our example, Yandex.Browser will appear.

1. Using the above link, panel or panel with bookmarks in a web browser, go to the home page of the domestic search engine.



2. In the upper right corner, find the item "Settings" and click on it with the left mouse button (LMB).



3. This action will expand a small menu in which you should select the last item - "Portal settings".



4. You will find yourself on the Yandex settings page. Make sure the tab is open "Search" shown in the image below and remove the section "Search Hints" checkboxes next to items "Show frequent queries" And “Show sites you visit frequently”.

   

   Note: If you wish, you can also clear your search query history by "Search settings" a separate button is provided.

   

5. After unchecking the items indicated above, click on the button located below "Save".



Returning to the Yandex home page or going directly to the search page, you will no longer see any prompts when entering a query.

When a user begins to enter a query in the Yandex search bar, the search engine shows several of the most popular queries starting with the letters already entered - these are search suggestions. Search suggestions help save time - you don't have to type the entire query. Yandex understands which hints to show, even if the user forgot to change the keyboard layout or made a typo.

Preparing hints

The list from which search suggestions are taken is formed after filtering the entire flow of user requests to Yandex (and also from the names of encyclopedic articles, musical works and other suitable content). Requests go through a dozen filters, each of which filters out requests based on several conditions. For example, it removes very rare queries or those containing profanity. Along with query filtering, typos are corrected. As a result, more than a hundred million requests remain.

Among the remaining queries, similar ones are searched to combine them into groups. For example, some users ask [gifts for March 8th], and some [gifts for March 8th]. The meaning of the request is no different, and when the user types only “gifts”, Yandex shows the most popular option. In this case, with the preposition “on”. Of course, if a person continues to write “gifts for”, a corresponding set of prompts will appear. They approach grouping queries very carefully - queries that seem similar to a machine are not always similar to a person. For two queries to be combined into one hint, they must not only differ slightly in spelling, but also lead to the same search results.

Since new popular queries are constantly appearing, the list of search suggestions is regularly updated - at least once a day. Requests that are no longer relevant are deleted.

For queries about events and incidents that have just happened and are of interest to a large number of users, Yandex has a “quick” list of tips. It is updated every half hour. Queries for this list are selected using a complex formula that takes into account how dramatically search interest in the topic has increased, how many news reports and blog posts have appeared, etc.

In some cases, already at the stage of typing a request, it is possible to say with a high probability that a certain site will be a good answer to the user. Then, among the search tips, a navigational one will also appear - the address of such a site. For example, based on the first letters of the query [Wikipedia], the first hint will be ru.wikipedia.org. When selecting a navigation hint, the user is immediately taken to the corresponding site.

Appearance of tooltips

While a user is typing a query, they are shown an average of ten sets of suggestions. Over the course of a whole day, Yandex shows hints to all users more than a billion times.

Like search answers on yandex.ru, search suggestions depend on the user's region. For example, when starting to write a request for [cinema] or [restaurant], a St. Petersburger or a Muscovite will probably have in mind establishments in their city. And they need tips for St. Petersburg and Moscow, respectively. Each region has its own list of search suggestions based on queries from that region.