Command-line hacking: displaying a weather summary

This is the first in what I hope will become a series of articles on doing off-beat and (I hope) interesting things with standard Linux command-line tools. In this post I'll give some ideas on retrieving a weather forecast summary from the BBC's RSS feed, and formatting it nicely. The idea is that you can just type, for example, weather at the prompt, and see a daily weather summary.

The basic strategy will be use curl to retrieve the weather forecast in RSS (XML) format, xmllint to extract the relevant parts of the document, sed to convert the text into groff format, then finally groff to format it, adding colour and emphasis. groff is highly configurable and produces nicely left- and right-justified text. It's easy to change this layout if required.

A quick-and-dirty alternative to using groff would be to use fmt, and then use sed to insert terminal escape codes to add the emphasis and colour. But this isn't very elegant.

Disclaimer: I don't claim that the way I go about solving this problem is the only way, or even the best. There are many ways to solve the same problem with shell scripting.

Prerequisites

To follow this example, you'll need curl and groff (which are part of the base distribution for many Linux variants) and xmllint (which probably isn't). On Ubuntu and similar, you can get xmllint by running:

$ sudo apt-get install libxml2-utils

Background

The basic URL for the BBC's 3-day forecast, in RSS format, is:

https://weather-broker-cdn.api.bbci.co.uk/en/forecast/rss/3day/NNNNNN

NNNNNN is a numeric code for the location. So far as I know, there is no published list of these location codes, and they don't correspond (to the best of my knowledge) to any geographic indicator. You can find the code for your location by looking at the BBC weather site and searching for the relevant name. The numeric code will appear in the URL. In the examples that follow, I'll be using the code for London, which is 2643743.

The following command will request the 3-day weather forecast for London, and dump it to standard out:

$ curl https://weather-broker-cdn.api.bbci.co.uk/en/forecast/rss/3day/2643743

If you examine the output, you'll see the following basic structure:

<rss ...> 
  <channel ...>
    <item>
      <title>Today: Light Cloud...</title>
      <description>Visibility: Good...</description>
      ...
    </item>
    <item>
      <title>Friday: Heavy Cloud...</title>
      <description>Visibility: Poor...</description>
      ...
    </item>
    ...

All RSS files are XML with this same layout -- only the content differs. In this case, there are three <item> elements, one for each day of the forecast. Each item has a title and a description which, to some extent, are pre-formatted display. Depending on the application, it might be sufficient to display only the "title" element -- the description is a bit wordy. Note that the title and description are intended to be displayed as provided. They do benefit from being formatted, though, and in my example I've added some text highlighting; still, it isn't easy actually to parse these text strings. Apart from anything else, the format tends to change, and relying on a particular entry (e.g., "Visibility") being present is a bad idea.

So our task here is to parse the XML, extract the title (and perhaps description), and format the results for display in the terminal. Here's what I'm aiming for:

Parsing the XML

It would be nice to be able to extract the relevant parts of the XML file using simple command-line utilities like grep and sed. Since the structure of the BBC's RSS is so regular, that's almost possible. We could, for example, just grep for lines that contain "title" and "description".

Unfortunately, there are other lines in the XML that contain these tags. Since they tend to occur in the same position, a quick-and-dirty hack would be to extract specific occurences by their position in the XML. However, this isn't very robust -- and remember that we're working with a file format that is not documented, and whose layout could change. We need to ensure that, so long as we get a valid RSS document, we can display something useful. Consequently, it's necessary to do real XML parsing to extract the data.

The xmllint utility makes this easy, so long as the parsing is not too complex. In this case we just want to extract certain elements, that have well-defined positions in the XML's tree structure. This will do the trick:

curl -s [BBC URL] \
   | xmllint --xpath rss/channel/item/title\|rss/channel/item/description 

The --xpath switch takes one or more paths, separated by a vertical bar (pipe). Be careful to prevent the shell treating the pipe as an instruction to perform a pipe operation: here I've escaped it with a backslash but, of course, there are other ways.

Note:
RSS XML files are usually encoded in UTF-8 format. This is a multi-byte text encoding, whose significance will become apparent later.

Converting the XML

The first task is to strip any remaining XML tags from the text. That's easy to do with sed

| sed -e 's/<[^>]*>//g'

sed operates on standard input, and outputs to standard output. It edits the text stream in accordance with the instructions provided on the command line. In this case, I'm using the s/search/replace/g instruction, which replaces all instances ('g' for 'global') of the pattern with "//" -- that is, nothing. The pattern is a regular expression; in this case it matches anything starting with "<" and ending in ">", except for the ">" symbol itself.

sed's pattern matching is "greedy" by default -- that is, it will match the longest sequence of characters that it can. This is why we need the slightly odd pattern described above, and it's a format that I'll be using repeatedly in this example. We only want to match the XML tags -- we don't want to match from the start of one tag, right to the end of the next, which is what will happen if we're not careful.

xmllint won't insert any blank lines between the XML sections it extracts. The text of the BBC weather feed contains no line breaks of its own, but we will need to break the lines later so they fit nicely on the screen. But first, we need to insert our own line breaks between each day's weather forecast, forming a blank line. That is, each days's forecast will be followed by a blank line. If we don't do that, when we format the text using groff, it will want to merge all the lines together, and then format the whole chunk. To change a line break into a blank line, I'm doing this:

| tr '\n' '#' \
| sed -e 's/#/\n\n/g' \

sed won't easily match on, or convert, end-of-line marks. So, instead I'm using tr to translate these into hash (#) characters, and then sed to translate these into pairs of newlines. Of course, this won't work if the text actually contains hash characters of its own -- you'd have to pick a different symbol.

My personal preference is to remove the Fahrenheit temperatures that the BBC feed provides, and just keep the Celcius. Your preference may differ, of course. The Fahrenheit temperatures appear in brackets: "(38°F)". To remove these:

| sed -e 's/ (.*F)//g' \

Again, I know that this pattern doesn't appear anywhere else in the text, so it's safe to match it this way. Note that I'm removing the extra space that appears in front of the opening bracket but, in fact, groff will do that later, anyway.

In my script I carry out a number of other simple transformations: shortening long words, that kind of thing. I'm not going to describe these in detail because they just use the same application of sed as the previous steps.. However, there are a few things that merit explanation.

In my script, I display the names of the days (which include "Today" and "Tonight") in red, to make the sections of the output easier to distinguish. Since I'll be processing the text with groff, I want to surround these words with groff colour codes. These are a bit arcance but, essentially, we need \m[name] to set the colour, followed by \m[] to set it back to the default.

My way of making this change isn't very elegant:

| sed -e 's/Tonight\|Today\|Monday\|Tuesday\|Wednesday\
\|Thursday\|Friday\|Saturday\|Sunday/\\m[red]\0\\m[]/g' \

Since the day name is always the first token on a line, there ought to be a more elegant way to match it. However, it isn't the first token on every line. All the other ways I found to match this text were no less ugly, so I'm keeping what I have until somebody suggests a better way.

Note that in the replacement text, \0 means "substitute the text that matched". Because the backslash character has a specific meaning to sed, when we want an actual backslash in the output, we have to write \\. It's this kind of fiddly that makes text-processing scripts so unreadable, but there isn't really an alternative.

Incidentally, sed can take multiple operations on the same command line. They are separated by semicolons. This is quicker, but a sed command with dozens of operations on the same line is profoundly unreadable (rather than just very unreadable). So I've used sequential invocations of sed with one operation each.

I've used a similar technique as the above to add other kinds of emphasis. To make text bold, for example, we surround it by the groff codes \fB and \fR ("format bold" and "format regular"). Again, we need to be careful about escaping the backslashes when writing these codes in a script.

Formatting the text

Finally, we're (almost) ready to pass the converted text, with its embedded formatting codes, to groff. First, though, we need to add a recognizable character or character sequence to the start of each blank line. I'm using "###", but anything that we can be sure won't otherwise appear is fine.

| sed -e 's/^$/###\n/' \

The reason for doing this is that groff always seems to pad its output with a bunch of blank lines -- I haven't found a way to stop it doing that when outputting plain text. We need to strip those blank lines, but we don't want to strip the blank lines that we inserted earlier to separate the different days. Marking the "real" blank lines with a specific pattern will allow us to do that.

The groff invocation looks like this:

| groff -k -Tutf8 -man \

The Linux version of groff can not (so far as I know) handle multi-byte input -- it assumes each character is a single byte. The BBC RSS feed does produce multi-byte characters. In particular, there is the "degree" sign, which isn't an ASCII character. The -k argument tells groff to run a pre-processor that converts multi-byte characters into groff formatting codes. Since most Linux systems use UTF-8 character encoding, and the RSS format is UTF-8, no other configuration is necessary to make this conversion.

The -Tutf8 switch allows UTF-8 output. -man is an instruction to format in a way that is suitable for a Unix man page; essentially, that means for output on a text terminal.

Tidying up

The final step is to remove the spare, empty lines added by groff, and then convert the "real" empty lines we marked using the tag "###" with real blank lines.

| sed '/^$/d' \
| sed 's/###/\n/g'

Further work

The simple script could be improved in many ways. Of course, the formatting of the output won't be to everybody's taste. It would be relatively easy to provide the script with command-line arguments so that you could select the forecast location, or change the amount of data that is displayed. The BBC also provides an RSS feed of current weather observations, rather than forecasts, and it could easily be parsed and formatted with the same approach -- and much of the same code -- I've described above.

Another, slightly useful enhancement would be to suppress the generation of terminal formatting characers if the output is not to a terminal.

Download

If you're interested, here is the Here is the complete source code for the script I've been describing in this post.