Two months of MapOSMatic improvements

March 8th, 2010

Since the december 2009 hackfest (reported here, here and here) and the official announcement of the improvements early january 2010, MapOSMatic has seen a sustained flow of interesting improvements. This post compiles the most important improvements that we have done recently, and some details on future improvements.

OcitySMap

Support for other countries has been added to OcitySMap, the Python module that actually generates the maps and the indexes. This support allows OcitySMap to generate correct indexes for other countries, by transforming street names such as “Calle de la Vida” into “Vida (Calle de la)”. This was made possible through the major re-organization of OcitySMap for internalization that took place during the december 2009 hackfest. So far, we have support for french, italian, spanish, catalan, brasilian portuguese, arabic, russian, dutch, crotian and polish, thanks to many contributors !

From a coding point of view, OcitySMap now uses the psycopg2 Python module to query the PostgreSQL database instead of pygresql. We made this change because Django, the Web framework that we use for MapOSMatic, already uses psycopg2. Using the same Python module to query the PostgreSQL database sounded like a logical choice.

MapOSMatic

MapOSMatic, the web front-end for OcitySMap, has also seen a large number of improvements. First, many translations have been added: german, italian, catalan, russian, arabic, brasilian portuguese, crotian, polish and dutch. For the Arabic translation, we have added Right-to-Left support in our web site design. Again, all the translations come from new contributors to the project.

The website has also been completely re-designed :

  • A brand new CSS, with a much cleaner design
  • The map creation page is now separate from the homepage, for more clarity
  • The selector for the city name in the map creation page now provides an auto-suggestion mechanism
  • The job and maps page have been completely re-designed, with a better design than the old table-based approach. The maps are now classified by first letter, and a search engine allows to quickly find existing maps.
  • A RSS feed has been added to keep track of map renderings taking place on MapOSMatic.org.

See Maxime Petazzoni’s blog post for details on these design improvements.

Coming improvements

We have already implemented further improvements, that are currently being tested on our development server :

  • A new rendering daemon. When a map request is entered on MapOSMatic, the map rendering doesn’t take place synchronously, since it is a quite consuming work. Map rendering requests are processed asynchronously, one after the other, through a daemon called maposmaticd. This daemon has been completely rewritten to use more Python-ic approaches, to better support timeouts (renderings taking too much time) and in the future to support parallel renderings.
  • Use of Pango for text rendering. Until now, OcitySMap was directly using Cairo for rendering text in the street index. While this was working fine for latin characters, it didn’t work for arabic or korean characters (and probably other non-latin characters). Therefore, we have rewritten the code that generates the index to use Pango on top of Cairo, so that all characters are properly rendered. This change needs a little more work but should be ready for production soon.

We have also identified one of the major reasons for which the rendering requests are sometimes so long to process. One of our geographic SQL query is taking a very long time, due to the absence of optimization. We are working on this issue with PostGIS/PostgreSQL experts, and hope to get down from 4 minutes 38 seconds for getting the list of streets for Toulouse to something around 3.2 seconds. As you can expect, we are really looking forward to making these new changes available to the MapOSMatic community!

Major improvements to MapOSMatic

January 4th, 2010

We have just sent the following announcement to OpenStreetMap lists, officially announcing the improvements made to MapOSMatic during the latest hackfest.


As a new year’s present, the MapOSMatic team is proud to announce that a new version of the maposmatic.org website has been put online, with major improvements over the initial version announced in September 2009.

For the record, MapOSMatic is a website that allows to generate city maps from OpenStreetMap data. Each map is divided into squares to easily find streets and is delivered with the corresponding street index.

The new MapOSMatic provides the following improvements :

  • Support for the whole world. Any location in the world can now be rendered on maposmatic.org.
  • OpenStreetMap database updated daily. Until now, the database had never been updated since the service was started in September 2009. Now, the geographic database used to render the maps is updated daily, providing maps with the latest contributions to OpenStreetMap. Each map contains the date at which it was generated.
  • Better city search engine. Thanks to Nominatim, we now provide a search engine that allows to find cities in a much more usable way: cities with the same name can be distinguished and the search works even when the city name is not completely correct.
  • Support for other languages. A few parts of the map rendering process is language-dependent and we now have the infrastructure to use language-dependent code. For the moment, we support English, French and Italian, but we are waiting for your contributions to support other languages. The website has also been translated to German and Italian.
  • Amenities in the index. In addition to the streets, we have added important amenities to the index: schools, town hall, post offices, places of worship, etc.

All these improvements are available now on http://www.maposmatic.org

You can follow the progress and improvements of MapOSMatic on our blog at http://news.maposmatic.org. MapOSMatic is of course free software, you can fetch its source code and contribute to the project, see http://savannah.nongnu.org/projects/maposmatic/.

Do not hesitate to send us your feedback, comments, suggestions and contributions to contact@maposmatic.org

Cheers,

The MapOSMatic team.


MapOSMatic hackfest, day 3

December 22nd, 2009

Yesterday was day 3 of the hackfest, and we of course spent our day working on MapOSMatic.

  • Support for amenities in index has been added. Now, in addition to the streets, you have the location of the town hall, police station and other public buildings, the schools, universities and other education buildings, and the places of worship. More improvements are needed in this area (translation of headings, support for amenities represented by polygons, fix incorrect grouping), but the general infrastructure is in place.
  • The main form to select the city has been further improved, with bug fixes, improved usability, etc. Said like this, it doesn’t sound like a big deal, but all this stuff is written in Javascript using the JQuery library and requires careful fine-tuning to be working. We easily spend a day on the same 20 lines of Javascript.
  • The language selection has been improved, with all french-speaking countries and all english-speaking countries added (we’re waiting for your contributions for other languages). The list of languages available is now automatically reduced to the languages spoken in the country in which the selected city is. For multi-lingual countries like Belgium, we have no way to know if the selected city is in the french-speaking area, the dutch-speaking area or the german-speaking area. Therefore, it is up to the user to choose between these languages. This selection changes how the streets are sorted and the list of prefixes that we consider before sorting.
  • The slippy map based on OpenLayers is now loaded only if bounding box mode is used. This allows a much quicker loading of the MapOSMatic homepage.
  • We have improved our server infrastructure to regularly update the coast lines, which are coming from a difference source than the planet OSM xml file. It allowed to fix rendering problems in island where we had ugly coast lines. These informations are now updated once a week.

We’re now starting day 4 of the hackfest. We will soon send out a notification to beta-testers so that they can test all these new features and report their comments and bugs.

MapOSMatic hackfest, day 2

December 21st, 2009

Yesterday was day 2 of the hackfest, and we did a number of improvements :

  • Cities can now be searched in an efficient way, thanks to the usage of Nominatim. The “Administrative city” field has been turned into a small search engine that returns, through Javascript, the list of cities matching the request. We spent quite some time understanding the results of Nomimatim and how they relate to the objects in the PostGIS database generated by osm2pgsql. We still need to fix a few issues with this Javascript-based search engine.
  • We used to identify cities by their name, but this of course doesn’t work since many cities have the same name around the world. So, now, rendering of city by administrative boundary are identified by the OSM id of the polygon in the PostGIS database. OCitySMap has been modified consequently, and jobs submitted through MapOSMatic are properly handled by maposmaticd.
  • The language used to generate a map can now be selected before generating the map, and several other improvements have been made on the internationalization. The main topic remaining is to pre-select the languages that are useful depending on the country in which the city being rendered is in.
  • The slippy map integration has been reworked to post-pone the loading of the tiles so that the homepage of MapOSMatic is much faster to load.
  • The documentation to install MapOSMatic and OCitySMap has been further improved.
  • Several bugs or tasks waiting in our bug tracker have been fixed, and some cleanup has been done on bugs that were already fixed by previous development

We expect to put the new version into production tomorrow, with the main achievement of having world-wide support, a better city search feature, and internalization for maps. Of course, we’ll need a lot of contributions for the internationalization support.

MapOSMatic hackfest, day 1

December 20th, 2009

The hackfest started just yesterday around noon with the arrival of six crazy hackers who will spend four days working on MapOSMatic. For the first day, we spent some time setting up MapOSMatic on the laptop of all developers and then started working on different topics :

  • A group started experimenting with Nominatim in order to improve the city lookup feature of the MapOSMatic website. This is very important for several reasons. First because for the moment our city lookup mechanism only finds the city if the entered name matches exactly the one in the database (spaces, dashes and accents included). And also because we have no way of letting the user choose between several cities of the same name, which is very important as we intend to put the worldwide database into production soon. The initial work we did is to add a small gateway to Nominatim in MapOSMatic (see this commit). The rest of the work is being worked on now.
  • Another developer worked on the infrastructure to allow the adaptation of OCitySMap to other languages: how the street names should be sorted, what are the common prefixes for streets, places, roads, etc. See this commit, this commit, this commit, this commit, this commit and this commit. Support is only for fr_FR and en_GB at the moment, and only at the OCitySMap level. Nothing has (yet) been implemented to select this from the MapOSMatic website, but it’s obviously on the TODO-list for this hackfest.
  • The documentation for installing OCitySMap and MapOSMatic has been improved and updated for newer distributions.
  • The german translation contributed by malenki has been added to the website and now allows to access the MapOSMatic website in German. Contributions for other languages welcome, contact us for details.

The previous hackfest took place in August in Toulouse (south of France) where we enjoyed a very nice and warm whether, wearing shorts, eating tomatoes and cucumbers straight from the garden. It was just a perfect summer hackfest. Now, we’re in Méridon (near Paris), where snow has fallen since Friday and was falling again today, so we are enjoying a perfect winter hackfest, with a nice tartiflette yesterday evening ! :-)

MapOSMatic hackfest next week

December 15th, 2009

The MapOSMatic application website was initially created in August 2009 by a group of free software enthousiasts during a one-week intensive hackfest. The same group of people is going to meet again for a 5 days hackfest next week and we plan to take advantage of this hackfest to work on the major issues of MapOSMatic :

  • Put the world, daily-updated, OpenStreetMap database into production, so that cities from all over the world can be rendered and up-to-date data is used
  • Improve the city selection, to make it fuzzy, and to support various countries
  • Improve the city index generation to support different languages
  • Improve the web site display of existing renderings

Of course, a lot more issues are registered in our task list and in our bug tracker, but we will very likely focus on the points mentioned above.

MapOSMatic, image of the week on OpenStreetMap main wiki page !

November 27th, 2009

MapOSMatic is currently being featured on the main OpenStreetMap wiki page. More precisely, a MapOSMatic rendering of the french city Saint-Pierre-du-Perray has been slightly reworked, nicely mixing the map and the street index on the same picture, is featured as the « Image of the week ». Thanks to the nice contributors who did this!

Saint Pierre du Perray

Saint Pierre du Perray

The description below the picture says that we only cover France, which is true at the moment. But we have a database ready with updated data for the whole world, and expect it to make it publicly available soon. At least, guaranteed before the new year !

MapOSMatic.org in french magazine SVM

November 9th, 2009

Shortly after the public announcement of MapOSMatic.org, the french magazine SVM published an article on its website, entitled « MapOSMatic.org, on-demand creation of city maps ». Of course, many other websites also relayed the announcement of MapOSMatic.org, but SVM also announced it in its printed version, through a small interview of Thomas Petazzoni in its october issue :

maposmatic-svmFunnily, the main title of the magazine was about the supposedly wonderful new features of Windows Seven. Something most OpenStreetMap contributors probably don’t care about :-)

Performance testings, world database in production soon

November 7th, 2009

As the tests on the more powerful server didn’t show a dramatic increase of performances, we decided to stay with our existing server. After the addition of an index by David recently, I’ve done a few tests on rendering duration in different conditions :

  • On the France-only database (our “production” database)
  • On the France-only database, with the daily diff being applied on another database
  • On the world database (our “development database”)
  • On the world database, with the daily diff being applied to this database

For each case, I’ve done the rendering of three different french cities (Rennes, Colomiers and Sanguinet) in PDF, PNG and SVGZ formats, and I’ve done each test twice.
Rennes

  • World-database, without daily udate process: 2 minutes 5 seconds, 1 minute 51 seconds
  • World database, with daily update process: 3 minutes 3 seconds, 2 minutes 10 seconds
  • France-only database, without daily update process: 1 minute 48 seconds, 1 minute 47 seconds
  • France-only database, with daily update process on the world database: 3 minutes 54 seconds, 2 minutes 25 seconds

Colomiers

  • World-database, without daily udate process: 37 seconds, 32 seconds
  • World database, with daily update process: 1 minutes 5 seconds, 39 seconds
  • France database, without daily update process: 31 seconds, 31 seconds
  • France database, with daily update process on the world database: 51 seconds, 37 seconds

Sanguinet

  • World-database, without daily udate process: 33 seconds, 39 seconds
  • World database, with daily update process: 49 seconds, 56 seconds
  • France database, without daily update process: 32 seconds, 36 seconds
  • France database, with daily update process on the world database: 42 seconds, 40 seconds

Conclusions

First, we can conclude that when the daily update process is not running, the rendering time for a given city is only slightly longer on the world database than on the France-only database. So, the index added by David is very efficient and solved our main issue.

Secondly, when the daily update process is running, the rendering time increases up to 50% in the worst cases. It’s a lot, but the daily update process is very I/O intensive so we expected a large impact on the rendering time. However, the rendering times remain acceptable, since the rendering is done asynchronously on MapOSMatic.org.

Therefore, I think we will soon put into production the world database, which will be updated every day with the OpenStreetMap data.

Improve city name lookups

October 13th, 2009

In the course of finding some way to “optimize” support for the planet on our server, here is an index which speeds things up greatly for city name lookups:

create index admin_city_names on planet_osm_line (boundary,admin_level,name) where (boundary='administrative' and admin_level='8');

Read the rest of this entry »