Archive for the ‘ocitysmap’ Category

End of March 2012 MapOSMatic hackfest

Sunday, April 1st, 2012

The hackfest is now finished! The last changes were committed by Thomas at 9:00 on ocitysmap module and at 9:30 on maposmatic module this April 1st.

MapOSMatic hackers at work. From left to right: Thomas, Étienne, Sylvain and Gaël

Overall we have committed 127 changes on the ocitysmap module (17848 lines added, 3713 lines removed) and 40 changes on maposmatic module (3532 lines added, 1544 lines removed) over the 7 days of the hackfest. Some of those changes were made at 2:36 in the morning.

Drawing, testing, designing, listing bugs, ...

Now, given a city name, we are able to automatically produce booklet maps, with a nice overview page, the map spread over several pages and an index referencing for each street or amenity the corresponding map page number and the square on this page. Each map page has pointers to map pages at the North, East, West and South of this map.

All those changes can be seen for France’s map on the development website (take care to empty your browser cache and reload the website from scratch).

Before putting them in production, we need to fix a few remaining issues (choice of scale on poster maps, a few typos here and there, take into account latest translations, …) and to wait for the end of the world OSM database import (started 7 days ago!).

And then you’ll can enjoy the new MapOSMatic! Have fun! :-)

(Part of) MapOSMatic team. Top: Frédéric, Gaël and Étienne. Bottom: David and Thomas

MapOSMatic hackfest, start of last day!

Saturday, March 31st, 2012

Once again, we have done significant progress yesterday.

Thomas worked on:

  • The front page for a multi-page map with a nice title, the map overview and the map information (last database update time, copyright, etc.);
  • A nice Donate page. We hope to be able to buy some hardware to increase the responsiveness of our server and to cover some of our expenses for such hackfest;
  • A small test suite. Not perfect but better than nothing. ;-)
  • The integration of all the work submitted by the three others of us.

Étienne worked on:

  • An overview map at the beginning of a multi-page map. It nicely shows all the individual pages on a small map of the city, with the corresponding page number for each one of them. Moreover, we now generates individual maps only for the area covering the chosen city, thus reducing the number of pages!
  • The index, taking care that listed streets are part of the chosen city and not adjacent ones;
  • Each map page in a multi-page map, putting a grey shade showing the limit of the chosen city.

Gaël, on his side, made several small but nonetheless difficult to achieve improvements:

  • He fixed a bug in the multi-page map index creation. All streets for letters considered the “same” in a given language (e.g. “e” and “é” in French) are now correctly put under the same index category;
  • He reduced the thickness of the grid on each individual map;
  • He added some missing street prefixes for French;
  • He fixed a lot of other small bugs;
  • He looked at the way we are projecting the map on a plane. In fact, in the future we might change the projection to avoid distortions on the map by leaving Mercator and choosing another projection (like the set of UTM ones).

On my side (David):

  • I improved the multi-page map street index so that it now wraps long names over several lines;
  • I re-implemented from scratch the B&W style sheet made by Sylvain in myself in order to have all of our changes at all zoom levels. There are still some bugs left, like remaining colours on tunnels.

The result of all this work can be seen on this new version of a multi-page map for Issy-les-Moulineaux. You can even test it on our development website (for France only, sorry).

Today, Frédéric is going to join us. We have to finish all the small details by the end of today!

MapOSMatic hackfest, start of day 7

Friday, March 30th, 2012

Yesterday, Thomas and myself made significant progress on the  ulti-page rendering engine! We are now able to produce a PDF file of a city on several pages, with a small map overlap between each page. We also produce the global index at the end that references the correct square on each page. You can see the current work in progress on this map of Issy-les-Moulineaux (PDF, 9MB).

On his side, Gaël continued his previous work on improving the scaling code of ocitysmap, our back-end rendering code until a suitable patch could be sent to our mailing-list. He then started to work on the Javascript code used on the client side of MapOSMatic. Apparently there are lot of small stuff to fix and doing this is not that easy. :-)

On his side, Frédéric worked on the style-sheets, trying to understand how they are organized and making improvements on the installation documentation.

Today, Étienne will join us again. We have still a lot of work to do before the end of this week! Stay tuned… ;-)

MapOSMatic hackfest, start of day 6

Thursday, March 29th, 2012

Yesterday, Gilles Lamiral, author of imapsync, visited us. Yes the one behind the original MapOSMatic idea!  We had a good time to discuss about MapOSMatic, OSM, Free Software, business models, pitfalls of modern Internet connections and much more.

Gaël has worked on scaling issues. At one point in the code we are rescaling the map to fit in a given paper size, with a given resolution. This part of the code triggers bugs, for example the badly placed one way arrows. Gaël has tried to improve that part of the code, removing unneeded scaling. Difficult stuff that is not finished yet but Gaël is making good progress.

Thomas and myself have worked on the multi-page renderer. Following on previous Étienne work, we are trying to add the ability to split a big PDF map into several pages with a common index at the end. Yesterday, we spent the whole day trying to split correctly the original map into smaller pages that overlap with a small margin. Gaël found our bug late in the night and now the computation seems correct.

BTW, a big thank to leaflet: it helped us produce some quick debug code to show the computed bounding boxes.

MapOSMatic hackfest, start of day 5

Wednesday, March 28th, 2012

On Monday and Tuesday, we have started to work on more difficult bugs and elaborated features.

Thomas has tried to find why the one-way street arrows on the MapQuest style sheet are wrongly placed. In fact, he found a bug in Mapnik2. But there are still some strange things occurring with this style sheet. :-)

Sylvain and myself (David), we have worked on a new “Black & White” style sheet. This style sheet is a derived version of original OSM Mapnik style sheet that contains less colours, for example on roads or buildings. This style sheet should produce much more readable maps when printing on black and white printers.

Étienne, Thomas and Gaël have started to work on an old feature request: maps cut into several pages, with a common street index. This feature is complex and difficult to implement, thus there is not much to show right now. Stay tuned! ;-)

MapOSMatic hackfest, day 3

Monday, March 26th, 2012

We are now starting the third day of the hackfest and some progresses have been made.

First of all, we have fixed small bugs that where hiding in the corners:

  • The display on thumbnails was breaking the website with Django 1.3, this is fixed;
  • The port number to access the GIS (Geographical Information System) is now a configuration parameter;
  • We use right and left arrows, displayed in bigger size, to navigate in the Create map dialogue;
  • We display a In progress icon when somebody is typing characters to look for a city name, giving a better feedback to the user on what the website is doing;
  • The user can now navigate with the Nominatim results, with Next and Prev buttons;
  • We display a message that explains why no Nominatim result has been found and how to fix it;
  • Hide the right arrow when the paper size is loading in the Create map dialogue;
  • When the same amenity appears several times in the same grid coordinate (e.g. several building of the same town hall), it appears only one in the index.

We have also started to work on small feature requests. Until now, we have implemented:

  • The display, on both the website and the generated maps, of the most recent date at which the OSM data has been updated;
  • The display of villages and hamlets in the index, a useful feature in rural areas.

You can see the all those features on the current development web site (for France only).

We are also working on the set-up of a new world OSM database in a suitable format for this new version of MapOSMatic. That way, we hope to deploy this new MapOSMatic to the world before the end of this week.

Recent updates in MapOSMatic

Saturday, March 20th, 2010
Saint Laurent en Grandvaux, Jura, Franche Comté, France

Saint Laurent en Grandvaux, Jura, Franche Comté, France

MapOSMatic recently passed an important milestone in its history. A few weeks ago, we served our 10,000th rendered map (Saint-Laurent-en-Grandvaux, Jura, France) ! We are very pleased with this accomplishment and feel it is important for us to continue improving the MapOSMatic service for our users. We are working on several objectives in parallel:

  • improving the user experience on the website, adding new features and more translations;
  • reducing the rendering latency to produce maps faster for our users;
  • improving the maps and indexes themselves.

Today we rolled out on MapOSMatic a few changes that have been brewing and undergoing extended testing on our development setup, and as I write this post I am pleased to see that we are making good progress on these three main directions.

First and foremost, I think the most important update today comes from the addition of a Spanish translation of the MapOSMatic website (and the corresponding capability to generate maps and indexes internationalized for Spanish-speaking countries), contributed by Julio Costa Zambelli, Sebastian Borgwardt and Jean-Guilhem Cailton. We hope the availability of MapOSMatic in Spanish will help the Chile earthquake disaster response teams.

Next, we restored the bounding-box area input fields for specifying an exact zone to render by the coordinates of its top-left and bottom-right corners. We also added a new feature to the slippy map: by maintaining the Control key pressed, you can draw a rectangle directly on the map to precisely define the limits of the area you wish to render!

Small part of Seoul, South Korea

Small part of Seoul, South Korea

This update allows integrates better support for some languages :

  • The index of the streets and amenities, as well as the map and index titles are now rendered through the Pango library, which allows to correctly render Arabic, Korean or Japanese characters
  • The index is now rendered right-to-left for RTL languages such as arabic. The first column is on the right, the last column on the left, the street names are on the right while the square locations are on the left.
  • We also fixed a font problem with our Mapnik installation allowing for rendering of non-latin characters in the maps as well.
Small part of the index for Seoul, South Korea

Small part of the index for Seoul, South Korea

And finally, as advertised in a previous post about MapOSMatic improvements, we are now using a new version of the rendering daemon (the software in charge of processing the rendering requests queue and creating the maps and indexes). This new daemon lays the groundwork for future concurrent rendering and rendering jobs distribution which will help reducing the rendering latency, when completed. It also has better management of obsolete renderings, with the direct impact of no longer listing hundreds, if not thousands of maps we no longer have the rendered files for.

I would like to conclude this post by thanking, again, all the amazing contributors to MapOSMatic for their continuous feedback, ideas, bug reports and their celerity in providing translations updates when we add new features to MapOSMatic!

Two months of MapOSMatic improvements

Monday, March 8th, 2010

Since the december 2009 hackfest (reported here, here and here) and the official announcement of the improvements early january 2010, MapOSMatic has seen a sustained flow of interesting improvements. This post compiles the most important improvements that we have done recently, and some details on future improvements.

OcitySMap

Support for other countries has been added to OcitySMap, the Python module that actually generates the maps and the indexes. This support allows OcitySMap to generate correct indexes for other countries, by transforming street names such as “Calle de la Vida” into “Vida (Calle de la)”. This was made possible through the major re-organization of OcitySMap for internalization that took place during the december 2009 hackfest. So far, we have support for french, italian, spanish, catalan, brasilian portuguese, arabic, russian, dutch, crotian and polish, thanks to many contributors !

From a coding point of view, OcitySMap now uses the psycopg2 Python module to query the PostgreSQL database instead of pygresql. We made this change because Django, the Web framework that we use for MapOSMatic, already uses psycopg2. Using the same Python module to query the PostgreSQL database sounded like a logical choice.

MapOSMatic

MapOSMatic, the web front-end for OcitySMap, has also seen a large number of improvements. First, many translations have been added: german, italian, catalan, russian, arabic, brasilian portuguese, crotian, polish and dutch. For the Arabic translation, we have added Right-to-Left support in our web site design. Again, all the translations come from new contributors to the project.

The website has also been completely re-designed :

  • A brand new CSS, with a much cleaner design
  • The map creation page is now separate from the homepage, for more clarity
  • The selector for the city name in the map creation page now provides an auto-suggestion mechanism
  • The job and maps page have been completely re-designed, with a better design than the old table-based approach. The maps are now classified by first letter, and a search engine allows to quickly find existing maps.
  • A RSS feed has been added to keep track of map renderings taking place on MapOSMatic.org.

See Maxime Petazzoni’s blog post for details on these design improvements.

Coming improvements

We have already implemented further improvements, that are currently being tested on our development server :

  • A new rendering daemon. When a map request is entered on MapOSMatic, the map rendering doesn’t take place synchronously, since it is a quite consuming work. Map rendering requests are processed asynchronously, one after the other, through a daemon called maposmaticd. This daemon has been completely rewritten to use more Python-ic approaches, to better support timeouts (renderings taking too much time) and in the future to support parallel renderings.
  • Use of Pango for text rendering. Until now, OcitySMap was directly using Cairo for rendering text in the street index. While this was working fine for latin characters, it didn’t work for arabic or korean characters (and probably other non-latin characters). Therefore, we have rewritten the code that generates the index to use Pango on top of Cairo, so that all characters are properly rendered. This change needs a little more work but should be ready for production soon.

We have also identified one of the major reasons for which the rendering requests are sometimes so long to process. One of our geographic SQL query is taking a very long time, due to the absence of optimization. We are working on this issue with PostGIS/PostgreSQL experts, and hope to get down from 4 minutes 38 seconds for getting the list of streets for Toulouse to something around 3.2 seconds. As you can expect, we are really looking forward to making these new changes available to the MapOSMatic community!

Performance testings, world database in production soon

Saturday, November 7th, 2009

As the tests on the more powerful server didn’t show a dramatic increase of performances, we decided to stay with our existing server. After the addition of an index by David recently, I’ve done a few tests on rendering duration in different conditions :

  • On the France-only database (our “production” database)
  • On the France-only database, with the daily diff being applied on another database
  • On the world database (our “development database”)
  • On the world database, with the daily diff being applied to this database

For each case, I’ve done the rendering of three different french cities (Rennes, Colomiers and Sanguinet) in PDF, PNG and SVGZ formats, and I’ve done each test twice.
Rennes

  • World-database, without daily udate process: 2 minutes 5 seconds, 1 minute 51 seconds
  • World database, with daily update process: 3 minutes 3 seconds, 2 minutes 10 seconds
  • France-only database, without daily update process: 1 minute 48 seconds, 1 minute 47 seconds
  • France-only database, with daily update process on the world database: 3 minutes 54 seconds, 2 minutes 25 seconds

Colomiers

  • World-database, without daily udate process: 37 seconds, 32 seconds
  • World database, with daily update process: 1 minutes 5 seconds, 39 seconds
  • France database, without daily update process: 31 seconds, 31 seconds
  • France database, with daily update process on the world database: 51 seconds, 37 seconds

Sanguinet

  • World-database, without daily udate process: 33 seconds, 39 seconds
  • World database, with daily update process: 49 seconds, 56 seconds
  • France database, without daily update process: 32 seconds, 36 seconds
  • France database, with daily update process on the world database: 42 seconds, 40 seconds

Conclusions

First, we can conclude that when the daily update process is not running, the rendering time for a given city is only slightly longer on the world database than on the France-only database. So, the index added by David is very efficient and solved our main issue.

Secondly, when the daily update process is running, the rendering time increases up to 50% in the worst cases. It’s a lot, but the daily update process is very I/O intensive so we expected a large impact on the rendering time. However, the rendering times remain acceptable, since the rendering is done asynchronously on MapOSMatic.org.

Therefore, I think we will soon put into production the world database, which will be updated every day with the OpenStreetMap data.

World-wide support in MapOSMatic, even more issues

Friday, October 2nd, 2009

We have a development server running on a database holding the whole planet (as of today). Besides requiring 9h/day to keep the DB updated, the following usage issues arise on that “planet” development instance:

  • looking up an administrative name such as “Paris” takes ages (typically 2 to 5 minutes). This is not really great: the rendering jobs are queued, but such queries are not queued and will run in parallel. Besides, we should not ask the user to wait 2 to 5 minutes before the server confirms that it’s queueing her rendering
  • when rendering (ie. ocitysmap), finding out the city limits/boundaries of a city takes ages too. This is probably due to the same “feature” as above, but this time the problem is less critical as we are in the “qeueued” side of the force
  • in overall, the whole rendering process takes around 8 times longer with a DB holding the planet, than with a DB holding just France. This was measured on “Rennes”: 10mn rendering vs. around 1.25mn.

The “funny” thing is that both the “mapnik” and street index rendering don’t seem to be that slower, but this needs to be confirmed by a more thorough profiling. If this proves correct, we have to find out what makes PostGIS so slow for the first queries. Is this our queries ? The structure of the DB which is lacking a few indexes ? And, in that case, how come mapnik rendering + street index seem almost as fast: is this because the previous queries did put most things in cache ?

But what a great pleasure to be able to map NYC, Palo Alto, !…

Unfortunately, this will be our privilege, since we are decidedly not planning to open this to the public… for the time being, that is: with our current server (a “modest” dual core x86_64 2GHz / 2GB RAM). Donations or support for a new server are welcome !