Oh, how easy it is to get things wrong

Many people living in London and south-east London were woken up in the early hours of the morning today by an "explosion". It turned out to be a sonic boom caused by two RAF Typhoons flying supersonic to intercept a plane that had lost communications with Air Traffic Control.

Several news services carried the story including the Guardian, which included this image in the first version of the article:

It wasn't long before numerous people on Twitter pointed out that the plane shown in the picture is not an RAF Typhoon but a Russian aircraft. The caption on the photo gave credit to the Royal Air Force/AP. Surely the Royal Air Force cannot have got it wrong? I ran an image search for Typhoon fighter planes to compare with the Guardian image and it most definitely is not a Typhoon.

I then ran a reverse image search on the photo in Tineye and found that it had first appeared in news articles in June earlier this year (2019).

An article from Business Insider had the same image but with the caption: "A Royal Air Force Typhoon from No. 11 Squadron, based out of Ämari Air Base in Estonia as part of 121 Expeditionary Air Wing intercepts a Russian Su-27, June 25, 2019. Royal Air Force". 

Another image search, this time on SU-2, seemed to confirm its identity and by now there were several tweets on Twitter from those who know about such things pointing out The Guardian's error.
I suspect that someone at the Guardian had run a search on the photos that they were allowed to use and that search picked up the word Typhoon in the description of the image. In their haste to publish the article they didn't double check what was actually depicted in the photo. An easy mistake to make if you are in a hurry and don't know that much about fighter planes.

The error has now been corrected:

Google offers to include missing search terms - sometimes

Google has been omitting terms from searches for several years.  For me, the matter came to a head wayback  in November 2011 (see Dear Google, stop messing with my search).  Many of has had noticed it happening for a while but what suddenly made it more frustrating was that one could no longer prefix a term with a plus sign to force its inclusion in a search. Furthermore, surrounding terms and phrases with double quote marks did not always work either.

Google's Dan Russell explained why in a comment to my blog posting:

"When you do a multi-term query on Google (even with quoted terms), the algorithm sometimes backs-off from hard ANDing all of the terms together. It’s a kind of “soft” backoff. Why? Because it’s clear that people will often write long queries (with anywhere from 5 to 10 terms) for which there are no results. Google will then selectively remove the terms that are the lowest frequency to give you some results (rather than none). Bear in mind that 99% of searchers have no idea why they’d want to hard AND, and just get frustrated when they get no results. The soft AND is a way to reduce the overall frustration and give the searcher something to examine (and with luck, a chance to reformulate their query)."

He added:

"But I see what you mean about wanting to know if there are NO hits to a given query. I’ll pass this information along to the Google design team and see if we can’t do something with this."

Well,  Google did do something about it and some weeks later Verbatim, which could be applied to your entire search and make Google run it without omissions or variations,  was added as a tool. The other option that existed then, and still does, is to prefix individual terms or phrases with 'intext:'.

If you did not use Verbatim you were still left guessing as to whether or not all of your terms or their synonyms were present in a particular document until you actually clicked on it and viewed it in its entirety.  About a couple of years ago, Google started to include information on omitted terms in the results snippets  by adding a "Missing: " statement underneath the entry.  At least we now had something to work with.  Google has now added a search option to it.  It started to appear 2-3 months ago, disappeared for a while, but now seems to be a permanent feature.  It enables you to tell Google that it must include the missing term. Let's works through the example that first alerted me to it: a search for broad beans called Eleonora and supplied by Tamar Organics.

Before you ask, the reason I did not go directly to the Tamar Organics website was because it was quicker to go via Google than to work through the seed supplier's site search and navigation system. Also, please note that if you try this search out yourselves you will probably get very different results. When we tried this in a workshop of 20 people we ended up with 11 variations on the theme!

First, the quick and simple approach of just throwing in a few terms:

broad beans eleonora tamar organics



The first two results were relevant and exactly what I was looking for,  but 8 results seemed a bit low especially as Google had indicated on the next two in the list that the term "eleonora"  was missing.  (We'll come back to the "Must include: " in a moment.) Going to the bottom of the results page there was the usual message that similar entries had not been displayed.



Erm... but, Google, you displayed 8 not 15 as you claim.  Let's play along, anyway, and repeat the search by clicking on the link Google gives us. This time I was given 11 results.  We know that Google often gets the count wrong when using the repeat search option but I still thought that the number of results was rather low if it was omitting terms.  What would happen if I decided to take Google up on its offer of "Must include: eleonora"? Two, three or perhaps just four results?  I clicked on the eleonora link and .... 20,700 results!



In the search bar above the results we can see that Google has put eleonora in double quote marks to force its inclusion.

The first three results were fine but when I looked in detail at the fourth document it was missing both tamar and organics, and there was no indication in the snippets provided by Google that these, or any other terms, had been omitted.

Going back to my first set of results and looking further down the list I saw that, as well as one from which eleonara was omitted, there was another that had left out both eleonora and tamar, and a third with just tamar missing.



If the "Must include:" option has more than one term, you can only choose one of them. You cannot have all of them.  Choosing tamar gave me  43,500  results but this time Google did tell me when eleonora was missing from the documents. Most of the results were totally irrelevant.

How would I normally deal with missing terms?  I generally start off with a quick and dirty search and, unless I am looking for a particular type of document such as a presentation or industry report, I don't always use advanced commands.  I just type in the separate words and in this case I did get what I wanted at the top of the page. But what if I hadn't?

I was interested in the variety of broad beans called Eleonora but Google was omitting it from some of the results.  I could have done what Google did and use quote marks around eleonora but my experience is that Google sometimes ignores those if the number of results is low. My usual strategy is to use 'intext:' before the missing word, for example:

broad beans intext:eleonora tamar organics

This gave me 18,400 results with, again, most of them missing one or more terms.

Deciding to trust Google not to ignore double quote marks I changed my search to:

"broad beans" "eleonora" "tamar organics"



This time it was just 3 results, and when I repeated the search to include the omitted results I saw 5 but nothing from the Tamar Organics website itself. The reason for this was the presence of the phrase "broad beans" in the search string.  Looking at the results in my very first  search, I saw that Google was picking up the phrases "broad bean" and "beans (broad)" so I was now missing out on the top and most relevant results. A reminder that one needs to think very carefully about how and in what order search terms may appear in documents before applying phrase searching.

For comparison I applied Verbatim to the original quick and dirty search and got 411 results.  The main problem with that set was that Tamar and Organics were appearing in the documents separated by several words or even sentences.  When I applied Verbatim to the search string:

broad beans eleonora "Tamar Organics"

I was presented with a respectable list of 18 relevant results.

So, is the "Must include:" option worth using? It is quick and easy to apply, especially on a mobile device and I suspect that is why it has been introduced. However, it all starts to get very messy and complicated  if you try to use it on subsequent sets of results.  When I'm searching on my laptop, or on a desktop, I sometimes try the link but if that set of results is disappointing  and Google drops a different selection of terms I go back to my practice of using intext and/or Verbatim.  I also try double quotes around terms and phrases but my experience is that that Google still occasionally ignores them. It is entirely up to you which approach you use. How well each works does vary from one search to another, and on whether or not you are allowing Google to adjust results according to your search history and behaviour.  The important thing is to be aware of the options available to you and to be willing to experiment.

Presentation: free search tools for research information

Edited highlights from my recent workshop on search tools for research information are now available. Please note that not all of the services, search tools, examples or issues covered in the workshop are included in this version.

Slides can be viewed on Slideshare  or authorSTREAM. 

Somebody, please put Google News out of its misery

I didn't think Google News (http://news.google.co.uk/) could get any worse but I was wrong. The previous revamp was bad enough: no more advanced search, useless and irrelevant personalisation options, and don’t even think about trying to set up sensible alerts. Alerts were never that good at the best of times but were not improved one iota by the changes. And then they altered the structure of the RSS feed URLs so that, supposedly, your existing feeds no longer worked. I don't know why, but my old feeds are still delivering news and contain better quality information than the new ones I set up. 

In the latest incarnation, Google News has lost most of my topics,  the "For You" is total rubbish as is "Local", you can no longer manage and personalise the topics  (although that didn't really work anyway), and the RSS feed buttons have gone. I can only assume that this is  all down to the real time AI/ML that Google recently announced was going to be used to organize the news. (The new Google News: AI meets human intelligence ).

Existing RSS feeds still work, though and you can create email alerts for a news search if you run it from within the general Google results page. Run your search in "All" and then click on the News link. There is a Create Alert button at the bottom of your results, but one wonders how long that will last.

Someone should put Google News out of its misery, close it down and leave news searchable via the link on the main page.

And they may as well ditch Google Finance as well. That is a  shadow of its former self : no more portfolios for monitoring stocks, no more historical data for viewing and download, no more news annotations on the price charts, and the comparison option only works for two stocks at a time. If you are interested in monitoring the stock markets or researching individual companies for free get thee hence to Yahoo! Finance (https://uk.finance.yahoo.com/).   There was some doubt over the future of Yahoo! Finance when Yahoo! was acquired by Verizon and became part of Oath but, charting oddities aside, there does seem to be some development going on. The new “Sustainability” tab for example shows environment, social and governance (ESG) ratings from Sustainalytics (https://www.sustainalytics.com/).  There was positive feedback on it from some business librarians who attended one of my recent workshops.

So many of Google's services are going from bad to worse to totally pointless and unusable. No wonder, then, that people are starting to look seriously at alternative resources.

SmugMug buys Flickr - should we stay or should we go?

So the wait is over. When it was announced that Verizon was to buy Yahoo! there was concern as to what was going to happen to Flickr. Yahoo! never did much in terms of developing Flickr and what it did do was rubbish. Trying to add the location of your photo is an interesting experience at the best of times. You might be able to pin it onto the map but the name of the place is all too often wrong. I used to spend half my time on Flickr manually changing the location - not something to be taken on lightly - but I generally don't bother now. It's not worth the effort.

Then there are the auto generated tags that Flickr adds to your photos without asking for confirmation. (Flickr pulls out all the stops with automatic tagging). These are sometimes relevant and it is helpful to be reminded of tags that might prove useful when searching, but the error rate is far too high to leave Flickr to its own devices when generating these. Deleting the oddballs after you have uploaded individual photos is not too onerous but checking a back catalogue of thousands of photos for rogue tags is not really feasible. It explains why Flickr search results often include photos that no way match your search terms.

It would also be nice if we could have interfaces with social media and mobile apps that actually work.

And finally, many of us are looking forward to not having to use a Yahoo! account to log in.

Unfortunately, the email that is hitting subscribers inboxes right now states:

"Nothing will change immediately with regard to your Flickr account. You will still access Flickr with your current login credentials and you will have the same Flickr experience as you do now."

They do, though,  go on to say:

"We will continue to work to make your Flickr experience even better."

Hmm. We shall just have to see if that is going to work out. In the meantime we have until May 25th, 2018 to decide if we want our Flickr account and data transferred to SmugMug. If we don't:

"you must go to your Flickr account to download the photos and videos you want to keep, then delete your account from your Account Settings by May 25, 2018. If you do not delete your account by May 25, 2018, your Flickr account and data will transfer to SmugMug and will be governed by SmugMug’s Terms and Privacy Policy".

So there we have it. I shall stay with Flickr/SmugMug for the time being and see how things develop. In any case, I shall be backing up my Flickr photos as usual just in case something goes seriously awry.

Brexit - sources of information

Please note: a regularly updated version of this posting is now on the main website at http://www.rba.co.uk/sources/brexit.htm 

Those of us living and working in the UK are constantly bombarded with news and information of varying quality on Brexit. I regularly run workshops on sources of business information and,  inevitably, these now include a section on the European Union (Withdrawal) Bill, commonly referred to as Brexit. One of the exercises I give those attending the workshop is to draw up their own individual list of resources that they are likely to use for keeping up to date, or as starting points for researching the topic. We then produce a combined list for the whole group.  I have listed below a selection of those resources, concentrating on the more general sources rather than industry specific sites that were mentioned in some of the sessions.  It is by no means a comprehensive list and this blog posting will not be updated,  but I have created a separate web page Brexit - UK withdrawal from the EU, which will be added to and amended periodically.

EU referendum results

Electoral Commission EU referendum results
The Electoral Commission is the independent body that oversees elections and regulates political finance in the UK. This page shows the voting totals and results by region and by area within that region. You can download the results data in full as a CSV file. There are also links to results visualisations, information on grants to designated lead campaigners, the Electoral Commissions assessment of the EU referendum question and their recommended amendment, and the voting guides.

EU Referendum Results - BBC News
The BBC referendum results page and linked pages presents the same information as the Electoral Commission but in a slightly different way. There are links to the BBC news stories and videos on and around the date of the referendum.

Results of the United Kingdom European Union membership referendum, 2016 - Wikipedia 
Another page showing the voting results in a variety of ways but in addition this one has links at the end to external sources reporting on the run up to the referendum and local press articles some of which show a breakdown of the results by ward.

News

Brexit: research and analysis - UK Parliament
"Research and analysis from Parliament's libraries and committees on how leaving the EU will affect different policy areas in the UK".
Brexit email alerts on updates and new content are available.

Brexit: All you need to know about the UK leaving the EU - BBC News
Background information on the what has happened so far, what is happening now, what has been agreed and what needs to be agreed. There is also a long list of FAQs (frequently asked questions), many of which cannot be answered yet but some possibilities are discussed.

The Guardian - Weekly Brexit Briefing
A very useful summary and update from The Guardian newspaper on what has been happening over the past week. You can sign up to receive the briefing by weekly email and there is also a weekly Brexit Means podcast.

General News Search

If you are interested in seeing articles that represent a wider range of viewpoints and opinions, run a search on Brexit in Google News and Bing News. As well as the national and regional UK papers, these will also pick up stories appearing in the press in other countries.

Legislation

European Union (Withdrawal) Bill 2017-19 - UK Parliament
Use this page to monitor the progress of the Bill through Parliament and see related documents such as:

• Full text of the Bill as introduced and further versions of the Bill as it is reprinted to incorporate amendments (proposals for change) made during its passage through Parliament.


• Tracked changes versions of the Bill


• Explanatory Notes


• Full list of amendment papers relating to the Bill.


• Public Bill Committee and report stage proceedings


• House of Commons Library and House of Lords Library briefing papers


• Will write letters (Questions put to government Ministers during debates on Bills may be answered by the Minister saying 'I will write to the Hon Member'. “Will write” replies are not published in Hansard but are placed in the Library of the House concerned and published on the Parliamentary website.)

Alerts on changes to the page, stage reached by the Bill, and new documents are available by email and RSS.

Blog | UK Constitutional Law Association
Affiliated to the International Association of Constitutional Law. The UKCLA blog provides analysis and comment on matters of constitutional law in the UK. Not suprisingly, many of the current blog postings cover some aspect of the European Union (Withdrawal) Bill.

Jack of Kent blog
"News and comment on law and policy, from a liberal and critical perspective". Written by David Allen Green who is a legal commentator at FT.com and a former legal correspondent of the New Statesman. Currently posting mainly about Brexit.

Public Law for Everyone – Professor Mark Elliott
Another source of comment and analysis on the EU (Withdrawal) Bill. Written by Mark Elliott, Professor of Public Law at the University of Cambridge, a Fellow of St Catharine's College, Cambridge, and Legal Adviser to the House of Lords Select Committee on the Constitution. The views expressed on this blog are in a purely personal capacity.

Google makes it harder to change location for country specific research

Google has made a major change to search and it does not bode well. Results are now based on your current location. So what's new?  Google has always looked at your location, even down to city/town level, and changed the results accordingly. That is fine if you are travelling and want to find the nearest Thai restaurant via your mobile, for example. Presenting a list of eateries in my home town of Reading is no good to me if I'm away in Manchester and getting very hungry!

The problems start if you are researching a person, company or industry based in a country other than your own - let's use Norway as an example - or just want the latest news from that country.  The trick used to be to go to the relevant country version of Google, in this case www.google.no, run your search and Google would give preference to Norwegian content. It is a great way to get alternative viewpoints on a topic and more relevant "local" information on a subject. Now, regardless of which version of Google you go to, you will see the same results tailored for your home location.

In a blog posting Making search results more local and relevant Google says:

Today, we’ve updated the way we label country services on the mobile web, the Google app for iOS, and desktop Search and Maps. Now the choice of country service will no longer be indicated by domain. Instead, by default, you’ll be served the country service that corresponds to your location. So if you live in Australia, you’ll automatically receive the country service for Australia, but when you travel to New Zealand, your results will switch automatically to the country service for New Zealand. Upon return to Australia, you will seamlessly revert back to the Australian country service.

This confirms that mobile search is what Google is concentrating on. After all it is, one assumes, where Google makes most of its money but it does not help professional researchers.

There is a way around it but it is rather long-winded. You need to go to Settings - use either the link in the bottom right hand corner of your Google home page or the one near the top of a search results page - and click on Advanced Search .



On the Advanced Search screen scroll down to “Then narrow your results by…” and use the pull down menu in the region box to select the country.



I ran a search on Brexit in google.co.uk, google.no and a few other country versions of Google. All gave me essentially the same results.

Using the region filter and selecting Norway as the country I am given the following by Google:



Notice, though, that Google is giving me English articles or English versions of them. Google has decided that I would prefer English articles and I have to scroll down to number 10 and beyond to see pages in Norwegian. To get a  broader view of what is being said in Norway about Brexit I have to go back into settings, click on Languages and choose Norwegian/Norsk.



Oh - and you get slight different results if you go through a VPN and set Norway as the country.

What worries me even more is that Google could do away with the advanced search screen and the region filter with it.

Google says:

We’re confident this change will improve your Search experience, automatically providing you with the most useful information based on your search query and other context, including location.

No, Google. You have just made things more difficult for those of us who conduct serious, in-depth research. The way I feel about this change at the moment is that if you were a person I would take a baseball bat to your head!

UPDATE: In response to David Pearson's comment and reminder below.
Including a site command e.g. site:no in the search works relatively well for this particular example (Norway) and gives good but slightly different results. It will, of course, miss Norwegian sites that are registered as .com or other international domains. The amount of overlap (or lack of it) will vary depending on the country. It's another one to add to the list of strategies, which I am sure will become longer,  for dealing with this problem.