Archive for the ‘database’ Category

New storage for MapOSMatic.org

Tuesday, May 27th, 2014

You probably noticed that our copy of OpenStreetMap database has not been updated for more than two months. In fact, our current storage space of about 500 GB was completely filled by the database. :-(

The good news is that we are currently in the process of changing that storage! Two new Samsung S840 Evo 1 TB disks have been ordered, that will quadruple our storage capacity to 2 TB. :-)

Once the disks are installed, we will need to make a new fresh import of OpenStreetMap database. We hope to do this import in hstore format that would allow us to bring new features to MapOSMatic.

Stay tuned!

MapOSMatic is back!

Friday, November 30th, 2012

Good news everyone: MapOSMatic is back!

Several months ago, our copy of the OSM database started to have major issues and got corrupted, unfortunately beyond recovery. We needed to re-import the full database. At the same time, we were having performance issues with the OSM data replication because the traditional, rotational the hard drive of the database server was not fast enough and we couldn’t keep up with the rate of updates to the OSM database. And last but not least, OSM was changing its license which had a strong impact on the data in our database too of course!

Therefore “The Plan” was to move our database server to a new machine that had SSD drives and import on this fast database server a fresh copy of the OSM database. As you probably discovered, it took us much more time than we originally planned ;-)

Today, we are very pleased to announce that the new database server is now up and running and it is fast! MapOSMatic is back on and once again available to the OSM community! :) And a big thank you to the FSF France for giving us access to a powerful server to host our OSM database.

Please enjoy our service! We have a lot of queued jobs since the summer that the rendering pipeline is now processing. It will take a few days before all of them are processed, but rest assured the server is working hard to catch up :) We will make sure to send a note when the queue is empty and new renderings can be processed as soon as they are submitted.

– The whole MapOSMatic team (with a special thanks to Maxime that does all the admin stuff!)

Update Dec 5th, 2012: we have processed all the pending renderings in the queue since July. You can now submit new map rendering jobs and they will be processed right away!

Service interruption

Tuesday, September 18th, 2012

Our OSM database is currently down therefore the MapOSMatic website no longer works. We currently don’t know how long it will take to restore the system.

Edit 2012-09-18 16:24: Our database is corrupted and PostgreSQL does not restart. It might take a long time to restore a new one.

End of March 2012 MapOSMatic hackfest

Sunday, April 1st, 2012

The hackfest is now finished! The last changes were committed by Thomas at 9:00 on ocitysmap module and at 9:30 on maposmatic module this April 1st.

MapOSMatic hackers at work. From left to right: Thomas, Étienne, Sylvain and Gaël

Overall we have committed 127 changes on the ocitysmap module (17848 lines added, 3713 lines removed) and 40 changes on maposmatic module (3532 lines added, 1544 lines removed) over the 7 days of the hackfest. Some of those changes were made at 2:36 in the morning.

Drawing, testing, designing, listing bugs, ...

Now, given a city name, we are able to automatically produce booklet maps, with a nice overview page, the map spread over several pages and an index referencing for each street or amenity the corresponding map page number and the square on this page. Each map page has pointers to map pages at the North, East, West and South of this map.

All those changes can be seen for France’s map on the development website (take care to empty your browser cache and reload the website from scratch).

Before putting them in production, we need to fix a few remaining issues (choice of scale on poster maps, a few typos here and there, take into account latest translations, …) and to wait for the end of the world OSM database import (started 7 days ago!).

And then you’ll can enjoy the new MapOSMatic! Have fun! :-)

(Part of) MapOSMatic team. Top: Frédéric, Gaël and Étienne. Bottom: David and Thomas

MapOSMatic hackfest, day 3

Monday, March 26th, 2012

We are now starting the third day of the hackfest and some progresses have been made.

First of all, we have fixed small bugs that where hiding in the corners:

  • The display on thumbnails was breaking the website with Django 1.3, this is fixed;
  • The port number to access the GIS (Geographical Information System) is now a configuration parameter;
  • We use right and left arrows, displayed in bigger size, to navigate in the Create map dialogue;
  • We display a In progress icon when somebody is typing characters to look for a city name, giving a better feedback to the user on what the website is doing;
  • The user can now navigate with the Nominatim results, with Next and Prev buttons;
  • We display a message that explains why no Nominatim result has been found and how to fix it;
  • Hide the right arrow when the paper size is loading in the Create map dialogue;
  • When the same amenity appears several times in the same grid coordinate (e.g. several building of the same town hall), it appears only one in the index.

We have also started to work on small feature requests. Until now, we have implemented:

  • The display, on both the website and the generated maps, of the most recent date at which the OSM data has been updated;
  • The display of villages and hamlets in the index, a useful feature in rural areas.

You can see the all those features on the current development web site (for France only).

We are also working on the set-up of a new world OSM database in a suitable format for this new version of MapOSMatic. That way, we hope to deploy this new MapOSMatic to the world before the end of this week.

PostGIS database is back online

Friday, January 20th, 2012

Our PostgreSQL/PostGIS database is finally back online and operational! We have rescheduled all the failed jobs since the beginning of the outage so renderings are generated for all of them.

Currently the database unfortunately has a 21 days replication lag with regard to current OpenStreetMap data. The replication process is back online as well but will take a while to catch up. We’ll keep you posted on the status of the replication.

Thank you for your patience and a big thank you again to the FSF France for giving us access to one of their server to host our database and for taking care of it when it goes down :-)

PostGIS database down

Monday, January 16th, 2012

The server hosting our PostGIS database, gracefully made available to us by the FSF France, is currently undergoing maintenance. This unfortunately means we can’t process any maps at the moment, but we’ve been assured that everything is put in motion to bring this server back online as soon as possible, hopefully in the next couple of days.

We’ll keep you updated on the status of the database server; in the meantime new renderings will be queued up and the failed renderings will be rescheduled to make sure all the requested maps are generated. It does unfortunately also means these renderings will represent outdated geographical data while the PostGIS database replication catches up.

Two months of MapOSMatic improvements

Monday, March 8th, 2010

Since the december 2009 hackfest (reported here, here and here) and the official announcement of the improvements early january 2010, MapOSMatic has seen a sustained flow of interesting improvements. This post compiles the most important improvements that we have done recently, and some details on future improvements.

OcitySMap

Support for other countries has been added to OcitySMap, the Python module that actually generates the maps and the indexes. This support allows OcitySMap to generate correct indexes for other countries, by transforming street names such as “Calle de la Vida” into “Vida (Calle de la)”. This was made possible through the major re-organization of OcitySMap for internalization that took place during the december 2009 hackfest. So far, we have support for french, italian, spanish, catalan, brasilian portuguese, arabic, russian, dutch, crotian and polish, thanks to many contributors !

From a coding point of view, OcitySMap now uses the psycopg2 Python module to query the PostgreSQL database instead of pygresql. We made this change because Django, the Web framework that we use for MapOSMatic, already uses psycopg2. Using the same Python module to query the PostgreSQL database sounded like a logical choice.

MapOSMatic

MapOSMatic, the web front-end for OcitySMap, has also seen a large number of improvements. First, many translations have been added: german, italian, catalan, russian, arabic, brasilian portuguese, crotian, polish and dutch. For the Arabic translation, we have added Right-to-Left support in our web site design. Again, all the translations come from new contributors to the project.

The website has also been completely re-designed :

  • A brand new CSS, with a much cleaner design
  • The map creation page is now separate from the homepage, for more clarity
  • The selector for the city name in the map creation page now provides an auto-suggestion mechanism
  • The job and maps page have been completely re-designed, with a better design than the old table-based approach. The maps are now classified by first letter, and a search engine allows to quickly find existing maps.
  • A RSS feed has been added to keep track of map renderings taking place on MapOSMatic.org.

See Maxime Petazzoni’s blog post for details on these design improvements.

Coming improvements

We have already implemented further improvements, that are currently being tested on our development server :

  • A new rendering daemon. When a map request is entered on MapOSMatic, the map rendering doesn’t take place synchronously, since it is a quite consuming work. Map rendering requests are processed asynchronously, one after the other, through a daemon called maposmaticd. This daemon has been completely rewritten to use more Python-ic approaches, to better support timeouts (renderings taking too much time) and in the future to support parallel renderings.
  • Use of Pango for text rendering. Until now, OcitySMap was directly using Cairo for rendering text in the street index. While this was working fine for latin characters, it didn’t work for arabic or korean characters (and probably other non-latin characters). Therefore, we have rewritten the code that generates the index to use Pango on top of Cairo, so that all characters are properly rendered. This change needs a little more work but should be ready for production soon.

We have also identified one of the major reasons for which the rendering requests are sometimes so long to process. One of our geographic SQL query is taking a very long time, due to the absence of optimization. We are working on this issue with PostGIS/PostgreSQL experts, and hope to get down from 4 minutes 38 seconds for getting the list of streets for Toulouse to something around 3.2 seconds. As you can expect, we are really looking forward to making these new changes available to the MapOSMatic community!

Performance testings, world database in production soon

Saturday, November 7th, 2009

As the tests on the more powerful server didn’t show a dramatic increase of performances, we decided to stay with our existing server. After the addition of an index by David recently, I’ve done a few tests on rendering duration in different conditions :

  • On the France-only database (our “production” database)
  • On the France-only database, with the daily diff being applied on another database
  • On the world database (our “development database”)
  • On the world database, with the daily diff being applied to this database

For each case, I’ve done the rendering of three different french cities (Rennes, Colomiers and Sanguinet) in PDF, PNG and SVGZ formats, and I’ve done each test twice.
Rennes

  • World-database, without daily udate process: 2 minutes 5 seconds, 1 minute 51 seconds
  • World database, with daily update process: 3 minutes 3 seconds, 2 minutes 10 seconds
  • France-only database, without daily update process: 1 minute 48 seconds, 1 minute 47 seconds
  • France-only database, with daily update process on the world database: 3 minutes 54 seconds, 2 minutes 25 seconds

Colomiers

  • World-database, without daily udate process: 37 seconds, 32 seconds
  • World database, with daily update process: 1 minutes 5 seconds, 39 seconds
  • France database, without daily update process: 31 seconds, 31 seconds
  • France database, with daily update process on the world database: 51 seconds, 37 seconds

Sanguinet

  • World-database, without daily udate process: 33 seconds, 39 seconds
  • World database, with daily update process: 49 seconds, 56 seconds
  • France database, without daily update process: 32 seconds, 36 seconds
  • France database, with daily update process on the world database: 42 seconds, 40 seconds

Conclusions

First, we can conclude that when the daily update process is not running, the rendering time for a given city is only slightly longer on the world database than on the France-only database. So, the index added by David is very efficient and solved our main issue.

Secondly, when the daily update process is running, the rendering time increases up to 50% in the worst cases. It’s a lot, but the daily update process is very I/O intensive so we expected a large impact on the rendering time. However, the rendering times remain acceptable, since the rendering is done asynchronously on MapOSMatic.org.

Therefore, I think we will soon put into production the world database, which will be updated every day with the OpenStreetMap data.

Improve city name lookups

Tuesday, October 13th, 2009

In the course of finding some way to “optimize” support for the planet on our server, here is an index which speeds things up greatly for city name lookups:

create index admin_city_names on planet_osm_line (boundary,admin_level,name) where (boundary='administrative' and admin_level='8');

(more…)

Latest benchmarks for planet incremental diff imports on a higher end server

Monday, October 12th, 2009

On the higher-end server running osm2pgsql with a 8G cache, the speedup ratio is at least 2.1 and seems to increase with the size of the diff. Considering the very few samples we have, it’s hard to tell the effect of the cache size increase. But the fact is that this higher-end server behaves better, a speedup factor between 2 and 2.75 can reasonably be expected. This would translate to a daily server downtime of around 4h for planet updates. Details follow.
(more…)

First performance results on the higher-end server

Sunday, October 11th, 2009

Thomas already posted the results of the initial planet import on the higher end server (see the maposmatic-dev mailing-list): 52 hours, which corresponds to a speedup factor around 2 compared to our current server. Our first incremental diff import (with a 1.5G cache) shows a speedup factor of the same order, which means that an incremental diff import would require around 4h10 server downtime every day. Details follow.

We’re currently running another incremental diff import on the higher end server, this time with a 8G cache… Let’s hope it’s getting even better.

(more…)