Beyond the Box

AIMS – Why you should consider a central data repository for the large enterprise

One of the most powerful features in AIMS is the ability to point it at pretty much any spatial data source and then diplay that data in a map. For one of our large enterprise customers this was fantastic as they had data being managed by a variety of in-house and external systems that they wanted to bring together into their AIMS solution. 

The problem we found with giving AIMS direct access from to another system's spatial database or SHP file store was the potential performance impacts on the source system.  Given that these systems ranged in business criticality we found that the Application Owners were reluctant to let AIMS have open slather access to their data, even if it was only for querying.  When it's all said and done, using AIMS it is possible to theme your spatial data in such a way that it will cause a significant database hit, just like you can with a bad SQL query.

So we ended up advising on an architecture something like this...

Notice that the external data sources supply data to the AIMS database or fileserver, essentially making the total AIMS solution a data wharehouse.

This gave us the added ability to 'process' data on the way in. 

We could also do things like joining data from various sources together selectively in our database - this doesn't work so well when set up through AIMS Studio.

We could also optimise the data for viewing - especially in the database where you can create both spatial and standard indexes to accelerate things like Layer Filters and Themes.

An architecture like this gives you great control over the data that AIMS is consuming.  The tricky bit is how do you keep data current?  But that's for another blog post!

Thematic Filters in MapGuide / Infrastructure Map Server for Performance

In this post I'd like to present what might appear as a rather un-intuitive performance tip for Autodesk Infrastructure Map Server / MapGuide: Under certain conditions, superior performance can be achieved by using layer theming to perform row filtering rather than filtering at the database level.

Traditional Database Filtering

Frequently when configuring map layers you will want to display only a subset of the records in a table. For example, if you have a Land Parcels table, you might have separate layers for different types of parcels (e.g. Residential, Commercial and Archived). Additionally, there may be an overlay layer for parcel text. Separate layers allows these layers to be turned on and off separately by users.

Figure 1 - SQL database provides filtering

 

We usually recommend implementing filtering of rows in a database table using the database itself. This can be done using database views (as in Figure 1 above) or using layer filters (layer filters are translated to additions to the where clause in the SQL statement generated by the FDO provider).

Typically this achieves best performance because the database engine is more effective at filtering through its ability to use indexes and other optimisations, plus this reduces the size of the dataset transferred from the database to the map server.

In this particular example, where the filtering is performed by the database as in figure 1, there will be 4 separate queries to the database; one for each layer.

Thematic Filtering

To implement Thematic Filtering, we have to take whatever filters were in the database views and layer filters and add to the layer's thematic expressions. If the layer was using basic styling previously, it will need to be converted to thematic styling. For layers already using thematic styling, each thematic style will need its expression modified to include the filter from the database view and layer filter.

From the point of view of an individual layer, this arrangement is inefficient because some of the records passed to the each layer are not displayed by that layer.

In this configuration all 4 layers access records directly the same table.

Figure 2 - Each layer has conditional theming to exclude some records

Since the set of records requested by each layer is identical, Infrastructure Map Server is smart enough to only retrieve the set of records once. The saving in having a single result set shared by all 4 layers achieves an improvement overall performance.

Actual Performance Gains

Up to now this discussion has been theoretical, so let us now turn to a production installation of Infrastructure Map Server.

In this particular system, there are 6 parcel related layers all referencing the same underlying table.

Using the Performance Report functions in Infrastructure Map Server Site Administrator, first introduced in 2013, we measured an impressive 82% performance improvement for scales below 1:10000, with overall layer redraw time for these zoom scales dropping from around 3 seconds to sub 1-second.

Figure 3 - Performance improvements from a production system

You can clearly see in the graph above that with zoom scales 1:10000 and above, thematic filtering becomes less efficient. The reason for this is that 4 of the 6 layers in question are not active at 1:10000 and above, and hence reducing the result set from the database becomes the more efficient approach. This particular system has Infrastructure Map Server tiling enabled for the higher zoom levels, and starts where thematic theming benefits end.

Conclusion

In this post we have considered a scenario where thematic filtering to exclude records received from the database achieves superior performance to traditional database filtering such as database views and layer filters, the latter translate to database filters through the FDO provider. We also consider performance from a production system where a significant performance improvement is evident at lower scale ranges. Finite and differing scale ranges for the various related layers result in poorer performance at higher zoom levels; and tiling was suggested as an effective technique at these higher zoom levels.

Component Library Updater Tool

Here's a handy little tool for developers of add-on and customisation projects. This tool is free of charge and can be downloaded using the link at the bottom of this page.

A lot of our development work in Autodesk Consulting helps our customers make better and more effective use of Autodesk and related products through custom add-ons and enhancements.  Most frequently, we use the functionality exposed by the application's .Net API. Some of our enhancements touch multiple Autodesk applications.

Autodesk has a yearly release cycle for much of the desktop range of products. In between yearly releases there can be one or more Service Pack releases. And in between Service Packs there can be hotfix patches. If you have a custom component for a product that uses the .Net API, then you will probably already know that each release, Service Pack and/or Hotfix could require that your component be re-built against the updated assemblies from the target application.

Introducing: Component Library Updater. This tool automates the tedious task of updating the API library files for one or more development projects.

Put simply, the tool looks for filename matches between an installed application's files and the files in your development code folders. When filename matches are found, the tool compares the files so that only files needing to be updated are displayed.

Some features of the tool are:

• Exclude pattern matching on the destination folders (i.e. your development code files) essentially allows you to target just the files in the 'library' folders.
• Files are first compared by version number and length, and only then by actual file contents. This, along with an optimised file difference engine, is why this tool has such a fast file analysis process.

I'll run through the use of the tool with a quick example of the tool being used for a set of components developed for Autodesk Map 3D. In this case, I am upgrading the components from 2013 RTM to 2013 SP1.

Source Path Specifications

This item specifies the locations and file types that will be used to source the updated dll files.

If you haven't already done so, install the target application and apply service pack, hotfix, etc. to get it to the version you are targeting. If multiple applications are involved, get them all installed and updated to the desired target version.

Source locations are specified by full paths and can include wildcards, but searching within these folders does not recurse subdirectories. Note also that if a file could be matched from more than one of the source paths, the tool will use the first path that contains the file.

You can use UNC paths here too, which means you can utilise an application that has been installed on another computer.

Fire up the tool and enter the source locations.

Target Root Folder and Exclude Pattern

These two settings define the set of folders that the tool will look for and update matching product dll's. The target root folder is where you checked out your source code for all the components you want to update. The exclude pattern allows you to exclude folders like 'bin', 'obj', etc. that typically contain dlls, but would be little value in updating with this tool.

At Autodesk Consulting our team typically keeps copies of the referenced product dll's with our source code. Frequently this is in a subfolder named 'lib', 'Library' or similar. This keeps our source code projects somewhat independent from what is installed on any particular machine.

Analyse

Once you have source and target settings entered, you are ready to perform an analysis run. This will populate the Items to Update panel, but no files will be modified at this point.

Note that you can run Analyse as many times as you like.  This can be useful to in tweaking the exclude pattern so that you are targeting only the folders you want.

There are two views of the items to update. The folder view is essentially a summary, and tooltips display the list of matched files for each item.

Switch to the file view for details of the files that will be updated, including version numbers before and after.

In the file view, tooltips show the full path of the source file that will be used. This can be useful because the same file may exist in multiple source locations.

Update

Once you are happy with the list of files to be updated, click Update to proceed.

You should get a confirmation message letting you know how many files were updated.

Download

Windows binary:   ComponentLibraryUpdater_1.0.0

Source (Visual Studio 2010):   ComponentLibraryUpdater_1.0.0_src

No warranty is provided with this tool, but if you find it useful please let us know.

Porting AIMS Themeing to a New Oracle Environment

In one of the AIMS applications I work on, we allow our customer to do most of the themeing in AIMS. They are more than happy to set up exactly how they want the data to look and how the users will view and interact with it.  Because the customer does this themeing against a 'development' database and this themeing is later ported to the 'production' database that the application actually uses, sometimes the tables used for the development themeing may be named differently in the production database.

What these differences essentially mean is that you can't just export an AIMS .MGP package from the source environment and apply it to the ptroduction environment due to the differences in table names and schema names. If you open the Layer for editing within AIMS Studio and try to change the Data Resource or Feature Class, AIMS Studio will reset all your layer definitions…

There are two ways to get around this…

Editing the AIMS package

When you use the AIMS Site Administrator to create a package, the resulting file is <File Name>.mgp

This is in fact a ZIP file that contains all the Studio themeing in XML format. The steps I suggest for editing this file are:

• Take a backup copy first, in case you accidentally invalidate the package!
• Rename the file to a .zip (eg. before: My_Studio_Themeing.mgp After: My_Studio_Themeing.zip)
• Extract the zip file to a folder
• Drill down through the folder structures and locate the definition of the layer you wish to change. It should be something like this:
      <Layer Name>.LayerDefinition_CONTENT.xml
      <Layer Name>.LayerDefinition_HEADER.xml
• Open the 'CONTENT' file with a text editor (such as Notepad++) and make the required changes
  This is likely to be one or both of:
     <ResourceId>XXX</ResourceId>
     <FeatureName>YYY</FeatureName>
• Save your changes and close the file
• Re-zip the entire folder
• Rename the .zip file back to a .mgp

You can then import the package using the AIMS Site Administrator to establish the new environment with the correct definitions.

TIP: When the package has been unzipped, you can do a 'Find' in Windows Explorer to locate all occurrences of layers with the definition you wish to alter. You can then select and open all of these files in Notepad++ and do a global search and replace, followed by a 'Save all...".

Editing Layer definitions with MapGuide Maestro

There is an Open Source alternative to AIMS Studio called MapGuide Maestro. You can find it here: http://trac.osgeo.org/mapguide/wiki/maestro

I don't profess to know much about Maestro, however I do use it to edit the XML definition of my Layers. You simply start up Maestro and point it at your AIMS server. Then go to your LayerDefinition and right-click to choose 'Edit resource with xml editor'…

You are then able to make the required changes, again likely to be one or both of:
    <ResourceId>XXX</ResourceId>
    <FeatureName>YYY</FeatureName>

Make sure you save your changes, then open in AIMS Studio again to make sure the definition is correct by previewing your Maps!

Hopefully these steps will make a potentially tedious task that much easier.

Remote debugging in ASP.Net with Reflector Pro, Part 1

Introduction

When debugging your application, there are often times when it would be helpful to get a better understanding of what is happening beyond the boundary of your application's source code. For example, your project may reference a 3^rd party dll, or maybe you want to step through code in one of the .Net framework dlls.

That's where Reflector Pro's ability to debug a dll comes in. This is a truly wonderful feature - hats off to the team at Red Gate!

This post, however, is not an introduction to Reflector Pro's dll debugging feature. What I am presenting here is a solution to remote debugging a 3^rd party dll in an ASP.Net application. This is not as straightforward as you might think, and indeed before I came up with this solution I had thought it may not be possible at all.

The Problem

The core problem is that the symbol files (.pdb) created by Reflector Pro are on the local machine, which is inaccessible to the remote server.

The Solution

Share the local pdb store.

The Solution (in more detail)

The best way to explain is with an example. Here I'm attempting to debug the open source component OSGeo.MapGuide.Web.dll on a remote server named 'web-server'.

In this case, the component is open source so you could go to the OSGeo website, download the source, build in debug mode, copy the dll and pdb to your server to get the ability to debug the dll. But, as you will see, the procedure I have outlined in this post should get you debugging the dll far more quickly, and doesn't require that you download the source code.

The first step is to have Reflector Pro generate debug information for the dll we are interested in. Open Visual Studio and select 'Choose Assemblies to Debug' from the .NET Reflector menu.

You can see in the screenshot below I have selected a dll directly from an administrative share on the web server.

Strong Name Signed dll.

If the dll you are trying to debug is strong name signed, you will probably get this prompt from Reflector requiring that a debugger signature is added and the assembly is regenerated.

I have prepared a separate post to show how you can deal with this situation in an IIS remote debugging scenario.

Fortunately, this OSGeo dll is not strong name signed, so we can proceed.

 

Populate Reflector Object Browser

If you don't already have a Visual Studio project that references the dll you want to debug, the next step is to create one. Technically this is optional, but it's the easiest way to get the Reflector Object Browser to list the dll we are interested in. I created a Class Library project.

Now switch to Reflector Object Browser. From here you can drill into the dll, open class files set breakpoints.

Now for the real crux of this solution...

First we need to determine the Reflector symbol cache location.  This can be seen in VS, Tools > Options > Debugging > Symbols dialog.

What we want is the path to the 'Cache' folder, not including the '0' subfolder.  Open in Windows Explorer and create a share of the 'Cache' folder (not the '0') folder. 

I named the share ReflectorSymbolCache. Readonly permission is sufficient.

Now go back the Visual Studio Debugging Symbols options dialog.  Add a new symbol file location that is the UNC path to the share you have just created, and add the subfolder so it matches the local path.  My local pc is called 'dev-pc' so the path I'm adding is:

    \\DEV-PC\ReflectorSymbolCache\0

As this is a network location, VS automatically configures a symbol cache folder for you, even though this won't be used in this setup.

 

Start Your (debugging) Engines...

Login to your web server and start msvsmon (Visual Studio Remote Debugging Monitor).

Back to Visual Studio again, open the Attach to Process dialog. Enter the remote debugging qualifier name to connect to the remote machine and find the IIS worker process w3wp.exe that is hosting the applicaton you want to debug.

In the screenshot below there is only one w3wp.exe process, but there could be more if there are multiple IIS application pools active. You need to determine which one is the correct one for the code you are trying to debug. Sometimes it's easier to simply attach to all w3wp.exe processes.

If no requests for the application have been made since IIS started, the application pool may not have become active and the w3wp.exe process you want to connect to may not be present. If so, launch a browser to start the application and refresh the Attach to Process dialog.

You should get a warning regarding the automatically created symbol cache location, but you can safely ignore it. The local path to Reflector's symbol cache will be used on the local machine; only on the server will the UNC path be used. Therefore, the temporary SymbolCache folder won't be populated at all.

To verify that symbols have been loaded for the dll you are trying to debug, open the Modules window in Visual Studio (Debug > Windows > Modules). If all has gone well, you should see 'Symbols Loaded' for your target dll.

At this point the debugger is attached and your breakpoints should be active, solid red dots.

Happy debugging!

Oracle data tips for AIMS: Dealing with a large database table

Working with Enterprise customers, I've dealt with some very large spatial datasets containing database tables that are not ideally set up for AIMS. I was once working with a huge table, both in the number of records it contained (over 16 million) and the number of columns (169 in total, 45 of which were spatial). After setting everything up in AIMS Studio correctly, the geometry still did not appear in my map. Checking the AIMS logs I saw an error along these lines…

Error: Failed to stylize layer: NODE
An exception occurred in FDO component.
RDBMS: GQL syntax error at line 1, near 'ION_ORI",NI_UUB.LOCATION,NI_UUB."INT_SPE'
StackTrace:

After some trial and error, I found that the reason for this issue was that my TABLE had too many COLUMN/s for AIMS to deal with. In AIMS Studio, everything looked OK (in terms of all the COLUMN/s appearing as attributes in my layer definition), the Oracle data connection displayed the TABLE correctly and all COLUMN/s and definitions appeared OK. The only indication of the existence of a problem was that nothing displayed in the map and the abovementioned error message was found when I checked the 'Error' logs.

So the tip is to include only the data you absolutely require available to AIMS. To achieve this I created a VIEW in Oracle that selected only the COLUMNs that I needed. Here's a sample statement for creating a VIEW:

CREATE OR REPLACE VIEW <VIEW_NAME>
AS
(
SELECT <COLUMN_2>,
<COLUMN_11>,
<COLUMN_16>,
<COLUMN_GEOM_2>
FROM <TABLE_NAME>
);

And don't forget to:

1) Include the geometry COLUMN/s in the VIEW that you will be themeing in Studio

2) Include all COLUMN/s in the VIEW that you will be using as filters or conditional themeing values in AIMS Studio (you may consider indexing the source TABLE COLUMN/s that are being used here)

3) Include the attribute COLUMN/s in the VIEW that you want your users to see

4) Include the PRIMARY KEY COLUMN if you want your data selectable in the map

 

How to Build MapGuide Fusion

As a developer, the use of the term 'build' here isn't that accurate; really what we are talking about is combining, minifying and compressing Javascript.

Unfortunately, an out-of-box installation of MapGuide Fusion 2012 does not include all of the pieces of the puzzle necessary to perform the build. So, the first couple of steps are to sort this out.

Step 1 – Obtain build.xml

 You will need to get your hands on the build.xml file. Ideally, get the build.xml file for the version you are running. You can try getting the file from the OSGeo source repository.

Place the build.xml file in the fusion folder.

Step 2 – Download Apache Ant

Download from here. Once downloaded, extract the zip file somewhere on your hard disk.  

Step 3 – Create config_dist.json 

All you have to do here is copy the config.json file.

copy .\fusion\config.json .\fusion\config_dist.json

Step 4 – Build it!

 Run ant and it performs the build according to the build.xml file

call ..\apache-ant-1.8.2\bin\ant.bat compress

That's it. Just make sure you have your index.html referencing fusionSF-compressed.js (or fusionSF.js if you prefer – this file makes it easier to debug)

You might be wondering whether all the 'source' files for this build are needed in a production deployment and the answer is they are not. I have a separate blog post here on how to remove these files.

Remove source files from a MapGuide Fusion production deployment.

After spending the last couple of weeks unsuccessfully trying to think of something interesting to blog about, today a colleague in the US responded to an email that I sent almost a year ago where I claimed to have prepared a script to remove un-needed files from a MapGuide Fusion production deployment. Great - an ideal blog post topic!

If you aren't already aware, a MapGuide Fusion installation includes both the individual source files (mostly Javascript files) and 'built' fusion files. The built versions combine all the individual files into a single file and this achieves a significant improvement in browser performance over the use of the individual files. Naturally, the fusion web page index.html defaults to using the 'built' version.

If you are interested in how to build the Fusion Javascript file yourself, see my other post on this topic.

To do the clean-up, create a batch file as follows.

set fusion_base=C:\Program Files\Autodesk\Autodesk MapGuide Enterprise 2011\MgeWebServerExtensions\www\fusion del %fusion_base%\lib\SingleFile.js del %fusion_base%\lib\fusion.js del %fusion_base%\lib\jxlib.uncompressed.js del %fusion_base%\lib\Error.js del %fusion_base%\lib\EventMgr.js del %fusion_base%\lib\ApplicationDefinition.js del %fusion_base%\lib\MGBroker.js del %fusion_base%\lib\Widget.js del %fusion_base%\lib\Search.js del %fusion_base%\lib\Map.js del %fusion_base%\layers\Layers.js del /Q %fusion_base%\text\*.json del /Q %fusion_base%\lib\OpenLayers\Lang\*.* rd %fusion_base%\lib\OpenLayers\Lang del %fusion_base%\configHeader.json del /Q %fusion_base%\widgets\*.js del %fusion_base%\layers\MapServer\MapServer.js del %fusion_base%\layers\MapGuide\MapGuide.js del %fusion_base%\layers\Generic\Generic.js del %fusion_base%\lib\jquery-1.4.2.min.js del %fusion_base%\lib\OpenLayers.js del %fusion_base%\lib\RectTool.js del /Q %fusion_base%\lib\excanvas\*.* rd %fusion_base%\lib\excanvas

Edit the path for the 'fusion_base' setting in the above to suit your environment.

Your source files are backed up in some fashion, right? Excellent – go ahead and run the batch file.

The only thing now is to ensure your index.html file references the newly built fusionSF-compressed.fs file.

Please note that at present, this has only been tested with MapGuide 2011.

Invisible Fusion Widgets for Autodesk Infrastructure Map Server

Autodesk Infrastructure Studio (which is provided free with Autodesk Infrastructure Map Server - AIMS) can be used to easily drag & drop your newly developed fusion widget so that it becomes part of your web application. The widget can be placed in any of the area of your web Layout like a Toolbar or a Task Pane.

But what if you don't want the widget to appear as a visible UI element in any part of the application? For example, you may have a special widget that simply performs tasks like looking for the incoming URL or performs some background tasks which need not be part of the UI.

So to include the widget as part of your application:

1. Simply edit the index.html file of your chosen template for the web application.
2. Include your custom widget there as shown in the example below.

Now the widget is part of your application and ready to perform the tasks that it is designed for.

For example if ACSpecialLink is the widget that is to be included, it can be added like this:

Using this approach your UI can be enhanced easily to meet your custom AIMS application requirements.

Oracle data tips for AIMS: Enabling Object Selection

I've come across situations where I've successfully themed up my Oracle data in Autodesk Infrastructure Studio, yet when viewing the resulting map on my browser, I'm unable to select any objects to view their attributes. This is even after setting the Layer as "Selectable" in Studio. I've found the reason for this to be that I need a Primary Key on my source data in Oracle.

Creating a Primary Key in Oracle

There are two ways of ensuring you have a Primary Key on your table. You can do it during the creation of your table, or by altering the table after it has been created. Here's the SQL for both scenario's:

CREATE TABLE <TABLE_NAME>
(
"<COLUMN_1>" NUMBER,
"<COLUMN_2>" VARCHAR2(20 BYTE),
"<COLUMN_3>" VARCHAR2(30 BYTE),
"<COLUMN_GEOM>" "MDSYS"."SDO_GEOMETRY" ,
CONSTRAINT "<PK_NAME>" PRIMARY KEY ("<COLUMN_1>")
);

 

ALTER TABLE <TABLE_NAME> ADD CONSTRAINT <PK_NAME> PRIMARY KEY (<COLUMN_NAME>);

The above will work if you are using the table directly or through a VIEW that is an exact mapping of the source table. In case you are using a VIEW that joins two or more tables together, you will need to make sure you have a Primary Key on the VIEW.

ALTER VIEW <VIEW_NAME> ADD CONSTRAINT <PK_NAME> PRIMARY KEY (<COLUMN_NAME>);

After this has been done, you should be able to select your objects in the AIMS map and see all the attribute information!