O'Really?

April 1, 2014

The Serene Scientists Serenity Prayer via Jon Butterworth

Filed under: data mining,engineering — Duncan Hull @ 2:38 pm
Tags: , , , , , , , , , ,
banksy church

The Church of Banksy

Whatever your religous preferences, the Serenity Prayer by Reinhold Niebuhr captures a certain wisdom about life in general. So it is good to see that physicist Jon Butterworth at UCL has adapted it [1] for scientists:

“Give me grace to accept with serenity the things that cannot be understood,

Data to investigate the things which can be understood,

And the Wisdom to know the difference.”

Amen!

References

  1. Jon Butterworth (2014) Giles Fraser says scientists are replacing theologians. Some thoughts on that The Gruaniad, 2014-03-31

August 3, 2012

Where did all the BBC programme metadata go? The Infax catalogue online

Filed under: data mining,engineering,informatics,publishing,search,technology — Duncan Hull @ 8:25 am
Tags: , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , ,
BBC Logo in QR Code

bbc.co.uk/programmes as a QR Code by /Sizemore/ Mike Atherton on Flickr available under a creative commons licence

Over at @BBCSport and @BBC2012 there are some Olympian feats of big data wrestling going on behind the scenes for London 2012 [1]. While we all enjoy the Olympics on a range of platforms and devices, a team of twenty engineers is busy making it all happen. It’s great that the BBC, unlike other large organisations, can talk openly about their technology and share hard-won knowledge widely.

Back in 2006 the BBC published another impressive application that allowed users to search and browse over 75 years of programme data. The programme was built from  metadata, not the actual audio and visual data from the TV and Radio, but the data that comes after-data, information about the programmes from an internal database known as Infax [2,3].

The web app was published at open.bbc.co.uk/catalogue and built by a crack team of experts led by Matt Biddulph @MattB (and including Tom Coates @TomCoates, Ben Hammersley @BenHammersley, Gavin Bell @zzGavin, Tom Loosemore @TomskiTomski and several others – see comments below).

It allowed users to find weird and wonderful things. For example, you could browse all the programmes that Alan Turing or Albert Einstein had appeared in or search for all the programmes with Jennifer Ehle. You could query it as well, to list all episodes of Dr Who in the order they were aired. It wasn’t so much Big Data as Big Metadata, [4,5] potentially useful for improving the viewing and listening experience of the audience.

At the time of its launch, Dave Beckett @dajobe blogged about it, Matt Biddulph wrote some release notes, Tom Loosemore said a few words, backstage clocked it and I scribbled some notes too. Being a proof-of-concept “experimental prototype”, the app eventually disappeared into the great bit bucket in the sky. The only visible trace of the catalogue today is the blog posts above and the message below which greets you when you visit the site:

“Thank you for your continued interest in the BBC Programme Catalogue. The BBC is now looking into how this data can be incorporated into its programme information pages.”

You can still get some BBC programme metadata from bbc.co.uk/programmes and bbc.co.uk/archive. Every programme has publicly available metadata but only a fraction of what was in the open catalogue. Although the app has gone, lots of the data must still be there somewhere. Take for example, the opening ceremony for the Olympic Games…

The metadata that is currently available

The metadata for each BBC programme can be found via its own page, so the opening ceremony programme has metadata available in xml and rdf which tells you several things including this synopsis:

“Coverage of the opening ceremony, which officially starts at 9.00, with the eyes of the world focused on the Olympic Stadium as the 30th Olympiad is officially declared open by Her Majesty the Queen. Film director Danny Boyle is set to produce a stunning cultural show ahead of the athletes’ parade, during which over 200 countries are expected to be represented. This is followed by the official opening, the arrival of the torch and the lighting of the cauldron.”

The metadata also tells you that this particular programme was presented by Sue Barker, Huw Edwards, Gary Lineker, Jake Humphrey and Mishal Husain. The Executive Producer was Paul Davies and there’s a bunch of other stuff: date of first broadcast, links to related information and clips but that’s about it.

The metadata that used to be available

The great thing about the open catalogue was that it went into lots more detail than above. So, for the Olympics ceremony, the participants in the programme would have been listed as Danny Boyle, Daniel Craig, Thomas Heatherwick, Elizbaeth II, Paul McCartney, Rowan Atkinson, Bradley Wiggins, Kenneth Brannagh, Steve Redgrave, J.K. Rowling and so on. For each contributor, you could see what other programmes they had been involved in, not just recently broadcast ones, but those going back 75 years. You could also see who had collaborated with who and when their first broadcast was and so on. It didn’t just document the über-famous people either, it went into just as much detail about other people you might not necessarily have heard of like Frank Cottrell Boyce, Callum Airlie and Jordan Duckitt. It was great stuff, but neither the archive or current programmes seem to have this level of detail.

Meta-conclusions

It’s a bit of a mystery where all the lovely BBC metadata went, it’s probably just sitting on some servers somewhere, inaccessible to the outside world. With my licence fee paying hat on, this seems a bit of a waste. I’ve asked everyone I know, including people at the beeb, but have drawn a blank. Most have shrugged their shoulders and pointed to the useful but slightly impoverished /programmes and /archive which is why I’m writing this post on t’interwebs.

Maybe the Olympic task of curating all that data makes it un-sustainable. Perhaps somebody decided there is no point competing with wikipedia where wiki-nerds curate programme data for free? It’s possible you can’t justify serving big metadata without giving the actual data (programmes) too? Maybe there’s a shiny new application in the pipeline to replace the catalogue, currently being worked on or an upgrade to @ArchiveAtBBC & @programmes so they include much more data. Could there be issues with publishing this kind of personal data on the web which meant the whole thing got canned? Nasty copyright issues could probably sink a project like this too. Who knows…

Does anyone reading this know the answers? If you do, I’d love to hear from you.

[Update, In October 2014 the Big British Castle launched their BBC Geome project at genome.ch.bbc.co.uk which covers much of the same data as infax, from 1923 to 2009. There’s a pretty decent Wikipedia article on the BBC programme catalogue and its ancestors, including infax]

References

  1. David Rogers (2012). Building the Olympic Data Services, BBC Internet blog
  2. Ant Miller (2012). Opening up the Archives: Finding content using metadata from Infax catalogue and Radio Times, BBC Research & Development blog
  3. Ant Miller (2012). Opening up the Archives: New kinds of metadata, BBC Research & Development blog
  4. Karen Loasby (2006). Changing approaches to metadata at bbc.co.uk: From chaos to control and then letting go again, Bulletin of the American Society for Information Science and Technology, 33 (1) 26. DOI: 10.1002/bult.2006.1720330109
  5. Andrius Butkus and Michael Petersen (2007). Semantic Modelling Using TV-Anytime Genre Metadata, Lecture Notes in Computer Science, 4471 234. DOI: 10.1007/978-3-540-72559-6_24

January 15, 2010

Bio2RDF: Large Scale, Distributed Biological Knowledge Discovery

Filed under: ChEBI — Duncan Hull @ 2:11 pm
Tags: , , , , , , ,

Bio2RDFMichel Dumontier was visiting the EBI this week, here’s the details of his seminar Bio2RDF and Beyond! Large Scale, Distributed Biological Knowledge Discovery (slides embedded below) for anyone interested who missed it:

Abstract: The Bio2RDF.org [1] project aims to transform silos of bioinformatics data into a distributed platform for biological knowledge discovery. Initial work focused on building a public database of open-linked data with web-resolvable identifiers that provides information about named entities. This involved a syntactic normalization to convert open data represented in a variety of formats (flatfile, tab, xml, web services) to RDF-based linked data with normalized names (HTTP URIs) and basic typing from source databases. Bio2RDF entities also make reference to other open linked data networks (e.g. dbPedia) thus facilitating traversal across information spaces. However, a significant problem arises when attempting to undertake more sophisticated knowledge discovery approaches such as question answering or symbolic data mining. This is because knowledge is represented in a fundamentally different manner, requiring one to know the underlying data model and reconcile the artefactual differences when they arise. In this talk, we describe our data integration strategy that makes use of both syntactic and semantic normalization to consistently marshal knowledge to a common data model while leveraging explicit logic-based mappings with community ontologies to further enhance the biological knowledgescope. Coupled with the web-service based Semantic Automated Discovery and Integration (SADI) framework, Bio2RDF is well placed to serve up biological data for prediction and analysis.

Some quick notes: Bio2RDF is currently indexing around 5 billion triples, and is built with the open source Virtuoso database. There are some scalability issues in making the system cope with up to a total of 15+ billion triples currently required. There is nothing in Bio2RDF yet that deals with the redundancy problem, e.g. “buggotea” and its friends.

References

  1. Belleau, F., Nolin, M., Tourigny, N., Rigault, P., & Morissette, J. (2008). Bio2RDF: Towards a mashup to build bioinformatics knowledge systems Journal of Biomedical Informatics, 41 (5), 706-716 DOI: 10.1016/j.jbi.2008.03.004

September 4, 2009

XML training in Oxford

Filed under: conferences,informatics,publishing — Duncan Hull @ 3:13 pm
Tags: , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , ,

XML Summer School 2009The XML Summer School returns this year at St. Edmund Hall, Oxford from 20th-25th September 2009. As always, it’s packed with high quality technical training for every level of expertise, from the Hands-on Introduction for beginners through to special classes devoted to XQuery and XSLT, Semantic Technologies, Open Source Applications, Web 2.0, Web Services and Identity. The Summer School is also a rare opportunity to experience what life is like as a student in one of the world’s oldest university cities while enjoying a range of social events that are a part of the unique summer school experience.

This year, classes and sessions are taught and chaired by:

W3C XML 10th anniversaryThe Extensible Markup Language (XML) has been around for just over ten years, quickly and quietly finding its niche in many different areas of science and technology. It has been used in everything from modelling biochemical networks in systems biology [1], to electronic health records [2], scientific publishing, the provision of the PubMed service (which talks XML) [3] and many other areas. As a crude measure of its importance in biomedical science, PubMed currently has no fewer than 800 peer-reviewed publications on XML. It’s hard to imagine life without it. So whether you’re a complete novice looking to learn more about XML or a seasoned veteran wanting to improve your knowledge, register your place and find out more by visiting xmlsummerschool.com. I hope to see you there…

References

  1. Hucka, M. (2003). The systems biology markup language (SBML): a medium for representation and exchange of biochemical network models Bioinformatics, 19 (4), 524-531 DOI: 10.1093/bioinformatics/btg015
  2. Bunduchi R, Williams R, Graham I, & Smart A (2006). XML-based clinical data standardisation in the National Health Service Scotland. Informatics in primary care, 14 (4) PMID: 17504574
  3. Sayers, E., Barrett, T., Benson, D., Bryant, S., Canese, K., Chetvernin, V., Church, D., DiCuccio, M., Edgar, R., Federhen, S., Feolo, M., Geer, L., Helmberg, W., Kapustin, Y., Landsman, D., Lipman, D., Madden, T., Maglott, D., Miller, V., Mizrachi, I., Ostell, J., Pruitt, K., Schuler, G., Sequeira, E., Sherry, S., Shumway, M., Sirotkin, K., Souvorov, A., Starchenko, G., Tatusova, T., Wagner, L., Yaschenko, E., & Ye, J. (2009). Database resources of the National Center for Biotechnology Information Nucleic Acids Research, 37 (Database) DOI: 10.1093/nar/gkn741

June 19, 2009

Nettab 2009 Day Three: Semantic Integration

Filed under: conferences — Duncan Hull @ 11:26 am
Tags: , , , , , , , , , , , , , , , , , , ,

Catania ElephantA brief report (well just some scribbled notes, bullet points and links really) on the third and final day of Network Applications and Tools in Biology (NETTAB) 2009 in Catania, Sicily. There was a special section on Methods and Tools for RNA Structure and Functional Analysis. Disclaimer: RNA mania isn’t really my thing – so the RNA presentations and papers are grossly under-represented in this mini-report (sorry).

  • Keynote: Semantically Integrated eCommunities in Biomedicine: Next-Generation Models of Biomedical Communication, Tim Clark Massachusetts General Hospital and Harvard Medical School, Boston. His presentation opened by asking: What do the following have in common?

    1. Alzheimer’s Disease
    2. Huntington’s Disease
    3. Nicotine Addiction
    4. Schizophrenia
    5. Bipolar Disorder
    6. Autism
    7. Parkinson’s Disease
    8. ALS (Amyotrophic lateral sclerosis)
    9. Neuropathic Pain
    10. Major Depressive Disorder
    11. Cancer (multiple forms)

    Answer:

    1. Highly complex disorders
    2. Much information, incomplete understanding
    3. Inadequate treatment options
    4. Huge cost in human suffering
    5. Multi-factorial causality
    6. Require multi-disciplinary collaboration for progress to understanding and cure

    Tim discussed using The Science Collaboration Framework (SCF) a reusable, semantically-aware toolkit for building on-line communities. These make heavy use of Open Linked Data, controlled vocabularies and  Drupal to build websites to tackle the above disorders. For example pdonlineresearch.org (Parkinson’s Disease), StemBook.org (Harvard Stem Cell Institute) and alzforum.org (Alzheimers) [1]. The controlled vocabulary and ontology approach works well for understood stuff (where named entities are known) but not so good at the outer boundaries of our knowledge. Reusable framework for building web communities, Uses shared ontologies/vocabularies, Open source, freely available.

  • Michaela Guendel (Leaf Bioscience) presented DC-THERA Directory: A Knowledge Management System to Support Collaboration on Dendritic Cell and Immunology Research,  using cell type ontology, dendritic cell ontology, chebi, obi. Project involves Andrea Splendiani, Ciro Scognamiglio and Marco Brandizi
  • GePh-CARD: an information exchange application for an Hub & Spoke Network for Skeletal Dysplasias was presented by M. Mordenti & L. Sangiorgi
  • Panel Discussion: Collaborative and Social Bioinformatics Research and Development: Why, When, Who and How? Alex Bateman, Tim Clark, Duncan Hull and all participants. This panel discussion concentrated on Who? (experts vs. non experts, crowds vs. individuals, how to motivate and reward people to contribute to online communities. community annotation of data only possible when curators cede control of data) and then Where? (open wikis vs. closed ones, private vs. public data, wikis often not suitable for highly structured data, centralised vs. distributed systems)
  • Keynote: Bacterial Phylogeny and Taxonomy in the High-Throughput Sequencing World, Gabriel Valiente
  • Magdalena Musielak (has worked with Piotr Byzia) presented RNA tertiary structure prediction with ModeRNA,
  • Olivier Perriquet presented Improved heuristic for pairwise RNA secondary structure prediction,
  • Giampaolo Bella talked about Analysing microRNA by Theorem Proving. qualitative logic proving before quantitative experimental measures e.g. “shall we go to restaurant” before “how much does it cost”?
  • Mapping miRNA genes on human fragile sites and translocation breakpoints Alfredo Ferro et al.
  • Keynote: Computational challenages in the study of small RNAs Doron Betel, memorial sloan-kettering cancer center
  • microrna.gr. a suite of web based tools for elucidating microrna function was presented by Giorgo L. Papadopoulous, DIANA bioinformatics lab, biomedical Science research center, Alexander Fleming, Vari, Athens, Greece
  • Last but not least there was miRScape: a cytoscape plugin to annotate biological networks with microRNAs

The Tenth NETTAB (2010) Workshop will be in Rome, where the theme will be Oncology Bioinformatics and will be held at the end of  May or beginning of  June 2010.

References

  1. Das, S., Girard, L., Green, T., Weitzman, L., Lewis-Bowen, A., & Clark, T. (2009). Building biomedical web communities using a semantically aware content management system Briefings in Bioinformatics, 10 (2), 129-138 DOI: 10.1093/bib/bbn052

Blog at WordPress.com.