O'Really?

February 18, 2013

Metadata is after-data: not just data “about” data

Filed under: Science,technology — Duncan Hull @ 11:08 am
Tags: , , , , , , , , , , , , , , ,
metacrap

Metadata gets many people hot under the collar, especially Cory Doctorow and Paul Downey.

Lots of people will tell you that metadata is “data about data”. While this definition is commonly used, it isn’t very helpful. Instead, it can be more useful to think of metadata as after-data. This, after all, is one the original meanings of the word meta (μετά) in the greek language.

Metadata makes the world go round

Ask any data wonk and they’ll probably tell you that metadata makes the world go round [1]. The internets is built on metadata. For example, the image in this blog post, you might search or browse to the data through it’s metadata not the data itself – the actual jpeg image file. The usual line is that the image metadata (title, author, date, tags etc) is data about data, e.g. it is data about the image.

Where you draw the line between data and metadata is a bit arbitrary. Is the author of the photograph data or metadata? It’s often difficult to tell because the defnition of metadata is so wishy-woshy, it’s almost meaningless. Some people call it metacrap, see picture on the right.

Afterdata: what comes afterwards

In Greek, μετά (meta) also means after. So metadata can be thought of as afterdata and this helps shed a little more light on what metadata actually is. Data is what you see or hear first. Metadata is everything that comes after. Simples!

It’s a small, possibly pedantic, distinction betweeen about-data and after-data but for my money, a potentially useful one. I wonder what the internets will make of it?

References

  1. Li Ding et al  (2004). Swoogle: a search and metadata engine for the semantic web Proceedings of the thirteenth ACM international conference on Information and knowledge management, 652-659 DOI: 10.1145/1031171.1031289

August 3, 2012

Where did all the BBC programme metadata go? The Infax catalogue online

Filed under: data mining,engineering,informatics,publishing,search,technology — Duncan Hull @ 8:25 am
Tags: , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , ,
BBC Logo in QR Code

bbc.co.uk/programmes as a QR Code by /Sizemore/ Mike Atherton on Flickr available under a creative commons licence

Over at @BBCSport and @BBC2012 there are some Olympian feats of big data wrestling going on behind the scenes for London 2012 [1]. While we all enjoy the Olympics on a range of platforms and devices, a team of twenty engineers is busy making it all happen. It’s great that the BBC, unlike other large organisations, can talk openly about their technology and share hard-won knowledge widely.

Back in 2006 the BBC published another impressive application that allowed users to search and browse over 75 years of programme data. The programme was built from  metadata, not the actual audio and visual data from the TV and Radio, but the data that comes after-data, information about the programmes from an internal database known as Infax [2,3].

The web app was published at open.bbc.co.uk/catalogue and built by a crack team of experts led by Matt Biddulph @MattB (and including Tom Coates @TomCoates, Ben Hammersley @BenHammersley, Gavin Bell @zzGavin, Tom Loosemore @TomskiTomski and several others – see comments below).

It allowed users to find weird and wonderful things. For example, you could browse all the programmes that Alan Turing or Albert Einstein had appeared in or search for all the programmes with Jennifer Ehle. You could query it as well, to list all episodes of Dr Who in the order they were aired. It wasn’t so much Big Data as Big Metadata, [4,5] potentially useful for improving the viewing and listening experience of the audience.

At the time of its launch, Dave Beckett @dajobe blogged about it, Matt Biddulph wrote some release notes, Tom Loosemore said a few words, backstage clocked it and I scribbled some notes too. Being a proof-of-concept “experimental prototype”, the app eventually disappeared into the great bit bucket in the sky. The only visible trace of the catalogue today is the blog posts above and the message below which greets you when you visit the site:

“Thank you for your continued interest in the BBC Programme Catalogue. The BBC is now looking into how this data can be incorporated into its programme information pages.”

You can still get some BBC programme metadata from bbc.co.uk/programmes and bbc.co.uk/archive. Every programme has publicly available metadata but only a fraction of what was in the open catalogue. Although the app has gone, lots of the data must still be there somewhere. Take for example, the opening ceremony for the Olympic Games…

The metadata that is currently available

The metadata for each BBC programme can be found via its own page, so the opening ceremony programme has metadata available in xml and rdf which tells you several things including this synopsis:

“Coverage of the opening ceremony, which officially starts at 9.00, with the eyes of the world focused on the Olympic Stadium as the 30th Olympiad is officially declared open by Her Majesty the Queen. Film director Danny Boyle is set to produce a stunning cultural show ahead of the athletes’ parade, during which over 200 countries are expected to be represented. This is followed by the official opening, the arrival of the torch and the lighting of the cauldron.”

The metadata also tells you that this particular programme was presented by Sue Barker, Huw Edwards, Gary Lineker, Jake Humphrey and Mishal Husain. The Executive Producer was Paul Davies and there’s a bunch of other stuff: date of first broadcast, links to related information and clips but that’s about it.

The metadata that used to be available

The great thing about the open catalogue was that it went into lots more detail than above. So, for the Olympics ceremony, the participants in the programme would have been listed as Danny Boyle, Daniel Craig, Thomas Heatherwick, Elizbaeth II, Paul McCartney, Rowan Atkinson, Bradley Wiggins, Kenneth Brannagh, Steve Redgrave, J.K. Rowling and so on. For each contributor, you could see what other programmes they had been involved in, not just recently broadcast ones, but those going back 75 years. You could also see who had collaborated with who and when their first broadcast was and so on. It didn’t just document the über-famous people either, it went into just as much detail about other people you might not necessarily have heard of like Frank Cottrell Boyce, Callum Airlie and Jordan Duckitt. It was great stuff, but neither the archive or current programmes seem to have this level of detail.

Meta-conclusions

It’s a bit of a mystery where all the lovely BBC metadata went, it’s probably just sitting on some servers somewhere, inaccessible to the outside world. With my licence fee paying hat on, this seems a bit of a waste. I’ve asked everyone I know, including people at the beeb, but have drawn a blank. Most have shrugged their shoulders and pointed to the useful but slightly impoverished /programmes and /archive which is why I’m writing this post on t’interwebs.

Maybe the Olympic task of curating all that data makes it un-sustainable. Perhaps somebody decided there is no point competing with wikipedia where wiki-nerds curate programme data for free? It’s possible you can’t justify serving big metadata without giving the actual data (programmes) too? Maybe there’s a shiny new application in the pipeline to replace the catalogue, currently being worked on or an upgrade to @ArchiveAtBBC & @programmes so they include much more data. Could there be issues with publishing this kind of personal data on the web which meant the whole thing got canned? Nasty copyright issues could probably sink a project like this too. Who knows…

Does anyone reading this know the answers? If you do, I’d love to hear from you.

[Update, In October 2014 the Big British Castle launched their BBC Geome project at genome.ch.bbc.co.uk which covers much of the same data as infax, from 1923 to 2009. There’s a pretty decent Wikipedia article on the BBC programme catalogue and its ancestors, including infax]

References

  1. David Rogers (2012). Building the Olympic Data Services, BBC Internet blog
  2. Ant Miller (2012). Opening up the Archives: Finding content using metadata from Infax catalogue and Radio Times, BBC Research & Development blog
  3. Ant Miller (2012). Opening up the Archives: New kinds of metadata, BBC Research & Development blog
  4. Karen Loasby (2006). Changing approaches to metadata at bbc.co.uk: From chaos to control and then letting go again, Bulletin of the American Society for Information Science and Technology, 33 (1) 26. DOI: 10.1002/bult.2006.1720330109
  5. Andrius Butkus and Michael Petersen (2007). Semantic Modelling Using TV-Anytime Genre Metadata, Lecture Notes in Computer Science, 4471 234. DOI: 10.1007/978-3-540-72559-6_24

June 28, 2012

Fußball ist ein einfaches Spiel, bei dem 22 Spieler gegeneinander spielen und zuletzt immer die Deutschen gewinnen

Filed under: football — Duncan Hull @ 3:59 pm
Tags: , , , , , , , , , , , , , , , , , , , , , , , , , , , , , ,

If you are not interested in Football Science [1,2], look away now. Normal service will be resumed shortly.

Professor Gary Lineker has an intriguing idea:

“Football is a simple game; 22 men chase a ball for 90 minutes and in the end, the Germans win.”

“Football is a simple game; 22 men chase a ball for 90 minutes and at the end, the Germans always win” --Gary Lineker

The earliest source I can find for this infamous quote is an article in the Berliner Zeitung from 1996 [3] –  in German, obviously. The English original is unfindable at the BBC where it probably came from. Strangely appropriate, nein?

So it’s Germany vs. Italy tonight; glück, buona fortuna. May the best team win (probably Germany).

References

  1. Hopkin, M. (2006). Goal fever at the World Cup, Nature, 441 (7095) 793. DOI: 10.1038/441793a
  2. Miller, P.S. (2006). The computiful game, Nature, 441 (7094) 784. DOI: 10.1038/441784a
  3. Anon (1996). EM-Zitate, Berliner Zeitung

September 10, 2009

Why don’t scientists share data?

Filed under: communication,publishing,Science,web of science — Duncan Hull @ 2:28 pm
Tags: , , , , ,

Sharing Ice Cream by ClappstarAs Vince Smith once put it [1] data are the fuel of Science:

“The fabric of science is changing, driven by a revolution in digital technologies that facilitate the acquisition and communication of massive amounts of data. This is changing the nature of collaboration and expanding opportunities to participate in science. If digital technologies are the engine of this revolution, digital data are its fuel. But for many scientific disciplines, this fuel is in short supply.”

Despite the importance of data, some scientists are often really bad at sharing it properly. So why don’t scientists share data?

Nature has a special issue dedicated to this topic published today at nature.com/news/specials/datasharing [2,3,4,5] which isn’t behind a pay wall (at the moment). It describes some of the technical and cultural barriers to data sharing. You should go and read it yourself if you’re interested, but here is a very brief and incomplete summary, with some extra points thrown in for good measure:

  • Some funding bodies do not adequately support the research projects they sponsor in sharing data properly, both before and after publication.
  • Many scientists lack awareness, incentives and knowledge of data sharing which can be compounded by a fear of being “scooped”.
  • Public databases, often a more natural home for data than traditional publications, are frequently undervalued by a publish or perish culture [6].
  • Traditional scientific publishing is frequently (and ironically) a really inadequate method for sharing data. Important data and metadata routinely gets damaged or destroyed in the process of publishing [7].
  • The technical infrastructure for long term data sharing either does not exist or is not understood by those who should be providing and using it. This can lead to empty archive syndrome.

These are some of the reasons that scientists don’t share data. Which raises the question, how do we get out of this mess? The special issue offers some solutions [3,4] to these cultural and technical problems, including the use of Creative Commons licenses. It’s good to see these important issues given a higher profile but we will probably be striving for better data sharing for many years to come.

References

  1. Vince Smith (2009). Data publication: towards a database of everything BMC Research Notes, 2 (1) DOI: 10.1186/1756-0500-2-113
  2. Anonymous (2009). Data’s shameful neglect: Research cannot flourish if data are not preserved and made accessible. Nature, 461 (7261), 145-145 DOI: 10.1038/461145a
  3. Schofield, P., Bubela, T., Weaver, T., Portilla, L., Brown, S., Hancock, J., Einhorn, D., Tocchini-Valentini, G., Hrabe de Angelis, M., & Rosenthal, N. (2009). Post-publication sharing of data and tools Nature, 461 (7261), 171-173 DOI: 10.1038/461171a
  4. Toronto International Data Release Workshop Authors (2009). Prepublication data sharing Nature, 461 (7261), 168-170 DOI: 10.1038/461168a
  5. Bryn Nelson (2009). Data sharing: Empty archives Nature, 461 (7261), 160-163 DOI: 10.1038/461160a
  6. Michael Seringhaus, & Mark Gerstein (2007). Publishing perishing? Towards tomorrow’s information architecture BMC Bioinformatics, 8 (1) DOI: 10.1186/1471-2105-8-17
  7. Duncan Hull, Steve Pettifer, & Douglas Kell (2008). Defrosting the Digital Library: Bibliographic Tools for the Next Generation Web PLoS Computational Biology, 4 (10) DOI: 10.1371/journal.pcbi.1000204

[CC-licensed picture of Sharing by ryancr, more commentary on this post over at friendfeed.]

March 16, 2009

Defrosting the Digital Slideshow

Filed under: biotech,communication,informatics — Duncan Hull @ 3:14 pm
Tags: , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , ,

Slides from the seminar today, for those that asked for them. Thanks to everyone who came, we had a good turn out, much better than expected.

Those Library and Institutional Repository people have asked for an encore too…

July 19, 2006

AAAI: Google and the Semantic, Satanic, Romantic Web

Filed under: biotech,google,informatics — Duncan Hull @ 11:07 pm
Tags: , , , , , , , , , , , , , , , , , , , ,

Google Blue YellowTim Berners-Lee delivered his one hour keynote at the AAAI’06 conference yesterday on the Semantic Web, after an introduction from Yolanda Gil. Tim gave an impassioned speech covering the last 16 years of the web and discussed the future of sharing data on the web using persistent URI’s and W3C standards like RDF and OWL. At the end of it all, there were some searching questions from Peter Norvig, Director of Research at Google Labs.

Peter opened his questions by saying.

Many people usually ask me, when I stand up and ask questions after keynote speeches at conferences:

“Peter, what do you have against the Semantic Web?”

Here is roughly what Peter said, the semantic web will never work because:

  1. People are stupid: Google has lots of experience of dealing with peoples stoopidity on the web. Many people don’t write well-formed HTML, they don’t run web servers properly and they keep changing what their URIs identify. It sucks, but this is the world, imperfect and messy and we just have to deal with it. These same people can’t be expected to use the Resource Description Framework (RDF) and the Web Ontology Language (OWL), which are much more complicated and considerably less fool-proof. (Perhaps you could call this the dumb-antic web?!)

    Tim replied that a large part of the semantic web can be populated by taking existing relational databases and mapping them into RDF/OWL. The structured data is already there, it just needs web-izing in a mashup-friendly format. (What I like to call the romantic web: people will publish their data freely on the web this way, especially in e-science for example. This will allow sharing and re-use in unexpected ways.)

  2. People are competitive: People working for commercial companies and market leaders can’t be expected to just put their raw data on the web as RDF/OWL, they have little interest in standards. This is how they make a living, beating their competitors and locking their customers into proprietary data formats, so they can keep selling them software / hardware to use their data. (Analagous problems in science, scientists can be reluctant to share and publish data, if someone else will make new discoveries with it and claim all the glory)

    Tim replied that most bookstores thought putting their stock levels and prices on the web was a bad idea as it would give sensitive information away to their competitors. However, they soon realised that this would allow their customers to search, browse and eventually (kerr-ching) buy their books.

  3. People cheat and lie: People lie about what their content is about, again, Google is on the receiving end of this. Cheats try to fool the PageRank algorithm by saying their web pages are about books or movies, when they are really about Viagra or Pornography. The same fate awaits RDF and OWL, cheaters will use ontologies to tell bare-faced lies about their data. (What I like to call the satanic web: people do evil things).

    Tim didn’t have any good answers to this, although later in the day there were some papers touching on the issue of Trust and Policies in the semantic web layer cake.

These lively debates are raging on un-abated, in the corridors, lecture theatres and bars. AAAI is now in full swing and its great to be here!

May 5, 2006

Bioinformatics at the BBC: Demonstrating the power of metadata

Filed under: informatics — Duncan Hull @ 10:02 pm
Tags: , , , , , , , , , , , ,
The BBC programme catalogue has recently gone online, and provides a demonstration of the applications that can be built using web technology such as Ruby on Rails, RDF, FOAF, web feeds, tag clouds and sparklines. This impressive online catalogue has:IMGP4697

Unfortunately this catalogue currently includes no data, only metadata at the moment, so there are no audio or video streams yet, as this is an experimental prototype. As mentioned earlier the catalogue is based on RDF which will no doubt please Semantic Webhead Tim Berners-Lee and allows the database to be queried with SPARQL. One of the brains behind this is Matt Biddulph.

I wonder if a similar application could be built using the UniProt protein sequence and annotation data in RDF or the data currently being produced by the W3C BioRDF subgroup? Compared to biological databases the BBC catalogue is relatively small, although there are no figures on the size of the catalogue, which has been extensively hand-curated by experts over the years. The ratio of metadata to data is probably different too, where a typical biological database might have lots of data (e.g. raw protein sequence data) but poor quality and a low quantity of metadata (interactions, structures, functions etc).

However, this catalogue is an interesting prototype, which is addictively fun to play with and might spark a few imaginations in the bioinformatics community.

[update: seeAlso Alf Eaton’s visual TouchGraph of BBC TV/Radio Collaborators which allows you to browse this data more graphically. Unfortunately, this fantastic BBC Database is not always online. This post was originally posted on nodalpoint with comments].

Creative Commons License

This work is licensed under a

Creative Commons Attribution-Noncommercial-Share Alike 3.0 License.

Blog at WordPress.com.