Graphing weather data
For the past few years I have been running a couple of weather stations. The data is logged in a MySQL database and the latest values are displayed on a website. Graphing was done by the proprietary software that came with the weather stations.
Recently some of the weather stations started to give me problems, resulting in my graphing being messed up. I decided to create my own graphs from the data in MySQL. Drawing graphs in real-time from MySQL is very intensive, so a better solution with caching is needed.
RRDtool is the de-facto standard when it comes to time-based graphing. It is widely used for weather data, and is lightweight enough to create graphs instantly when a webpage is loaded.
By scheduling a Python script every minute, I pull the latest data from MySQL and add it to separate RRD's for every sensor. RRD automatically does averaging over certain time periods to make the graphs look good. This works great for sensors like temperature, humidity, barometric pressure and wind speed.
Wind direction on the other hand is a little more difficult. By averaging all wind direction measurements, the resulting graph will always output a line around 180 degrees (South). This is because RRD has no concept of degrees, and a number around 0 is not at all close to a number around 360. A direction just west of north (+-350 degrees), and a direction just east of north (+-10 degrees), will thus provide a direction around south (180 degrees).
To work around this problem is not so easy, but mathematics has a tool we can use. We can break a direction up into two components describing the angle. An east-west, or x component and a north-south, or y component.
x will be the sine of the angle, and y will be the cosine of the angle.
This computation is done in Python.
The x and y components are then added to a RRD where they are averaged for certain periods.
When the graph of the wind direction is drawn, the averaged x and averaged y is taken and the angle is computed:
This is done by RRDgraph's reverse polish notation parser.
If the answer is a negative angle, 360 degrees are added to put it in the correct range for wind direction measurements. The result of this method can be seen here on my weather website.
A strange and unexplained phenomenon is seen on the longer term graphs. The average direction always tends to approach 90 degrees. If this is an error, I do not know. Seeing as the general accepted average direction in and around my city is south-east, it can quite easily be an error.
If anyone has an idea why the average is around 90 degrees, please leave a comment.
The problem that I describe above was fixed after further investigation of the code I use. Both Python and PHP's trigonometric functions use radians and not degrees. Being used to measure wind in degrees I totally forgot this, and used degrees in the functions. The results were of course totally wrong. After changing the code to first convert to radians, the resulting graphs look much more believable.
Today someone contacted me to ask if I can tell them what is the prevailing wind direction in Stellenbosch. It gave me a reason to do some analysis in the data and the results are quite interesting.
Firstly I ran a SQL query to count how many times I had a sample for every unique direction. This result set was then exported to a spreadsheet, where the directions were aggregated in the 16 wind sectors. Below you will see two graphs from this aggregated data.
It is interesting to see the prevailing wind being north-east, while the second most common direction is exactly the opposite, south-west.
Comparing the directions of the three datasets I have, one gets the following graph: