That clip is the exact reaction I had when I walked into my office last Monday morning. Beginning last week, we got absolutely crushed/overwhelmed with traffic. A site that we developed and hosted was suddenly 'discovered' by the media (fox, associated press, USA Today, front page of DIGG, etc). Check out the sheer number of stories on it on google news here It continues today, like right now....currently our site is featured on the front page of CNN. Its a site that shows HIV disease prevalence across the country by county (and in some cases zip code) maphiv.org. I will warn you that registration (its a quickie) is required if you are interested.

Its an ArcGIS Server javascript api based site running on 9.3.1 using msd's & mxd based services with multi layer caches. We plan an extensive series of posts when things calm down a bit that will focus on the many lessons that we learned from this overwhelming amount of traffic.

QUICK UPDATE 12 noon edt: Our ArcGIS server is taking close to 20 requests per second and still holding strong.

Hey folks...Just wanted to share a site that is now using our remote cache hosting service to serve cached tiles.

I previously showed several Javascript API sites uing this service, but this one is using Flex api for ArcGIS Server.

This really frees up bandwidth to let the dynamic layers be served quickly. Those aerial images can be very bandwidth intensive, so hosting these remotely them makes a lot of sense when bandwidth is limited, or you have a high volume site.

Here is is: Santa Rosa Planning and Zoning Viewer. The layer that is being served remotely is the 'Aerial Photos' layer. If you are a developer geek like me, check out the site with firebug enabled so that you can see the Aerial imagery being served remotely - meaning a different location that the ArcGIS server itself. Cool stuff.

So, lets begin this discussion of Geoprocessing with ArcGIS Server that I promised. I guess the first question is what is geoprocessing and why should I even care?

The big promise of ArcGIS Server has always been this: "Whatever you can do in ArcMap and ArcCatalog, you can now do with ArcGIS Server". I have often heard ESRI reps say 'You can even rebuild ArcMap if you want to'. Of course, no one would ever want to do such a thing, but wouldn't it be nice to have all that functionality available to you in a web based application?

In the old days of ArcIMS and MapObjects IMS we were pretty limited in what types of geoprocessing we could do on the web. I can recall all sorts of crazy tricks with buffers that we used to write to really push ArcIMS to its limits. But, at the end of the day, it was still just a buffer, and there is only so far it will take you. How about a select by location, or a union, or a clip? Sorry folks, you were out of luck if you needed to do that on the web. But then, ArcGIS Server came around, and suddenly doing real geoprocessing became a reality.

So, what is a geoprocessing task? In short, it is a set of server side tools called from a web based client. Users create a model using Model builder, and then publish that model to the web as a geoprocessing task. The output parameters of the model are then streamlined to your web application.

Lets take for example a recent Flex application that I wrote (Check it out here). The purpose of this application is to help citizens locate their property and then return information about the zoning on that property. It sounds easy enough, right? I'll bet you are thinking, why not just do an Identify, and return results from the zoning layer? Ah, if it were only that easy.

There can be many, multiple types of zoning found on a single parcel, so a simple identify by point will not work. We need to do a spatial analysis to get the property boundary that the map click point or address point resides in. Once we get the property boundary, we need to do a second spatial analysis to determine which zoning layers intersect that property boundary.

So, how can we use a geoprocessing task to help solve this problem? I started by simply going into ArcMap and writing down the steps that it takes to get the results I need. Then, I used model builder to string together these tasks into a single workflow. Finally, I published this model to ArcGIS Server so I could access it from the web.

The, the final product takes in a point, selects the containing parcel for the point, and returns the zoning types that are present in the parcel.

Now, this is a pretty straightforward example. We could take this example much further. We could actually clip that zoning layer and determine precise percentages of each zoning type on that parcel. For parcels with multiple structures, we could determine the zoning district for each structure. List goes on.

This scenario, while certainly not impossible to do by conventional means (thinking FIND, QUERY, and IDENTIFY tasks), becomes much, much easier to solve by using a geoprocessing task. It also combines all the steps into one task. Instead of writing a complex combination of code to perform these steps in javascript or actionscript (flex) on the client, I have created a new, reusable tool that does all of these steps for me, serverside.

I hope this clears up what a geoprocessing task actually is and why you should care. Next up in the series, we'll tackle the technical aspects of actually creating and executing one from your web application.

Everyone always asks for an excuse to visit Charleston... well here is your chance! ROK is hosting two ArcGIS Server training classes this August at College of Charleston's GIS Lab.

Administering ArcGIS Server
2 Days (Monday, August 17th and Tuesday, August 18th) $850.00
Training will cover all the steps you need to get ArcGIS Server up and running in your organization. In addition, it will go into some of the more advanced functionality ArcGIS Server has to offer including building optimized map cache and publishing geoprocessing tools to the web. Here is a link to the outline for Administering ArcGIS Server:
http://www.roktech.net/training/Administering.pdf

Developing Applications for ArcGIS Server
2 days (Wednesday, August 19th and Thursday, August 20th) $850.00
Training will give you the tools and knowledge to start developing fast light weight mapping applications that users are now expecting. ArcGIS Server 9.3 offers new APIs that allow developers to get their data on the web in a quick, stylish, and developer friendly interface. We will cover the JavaScript API, Flex API, and briefly discuss the WebADF. On day two of this training, we will set aside time for you to apply the skills you've learned to start creating or continue developing applications for your organization. ROK developers will be there to help you along the way. Here is the link to the outline for Developing Applications for ArcGIS Server:
http://www.roktech.net/training/DevelopingApplications.pdf

For more information or to sign up for a class, please contact training@roktech.net

I hope to make this an ongoing series about utilizing Geoprocessing on the web. There is so much information to cover...I cant possibly cover it all in a post or two.

People that know me and perhaps have read this blog over the years know that I used to be a ArcGIS Server hater. We have been working with it since its first beta at 9 and had always found it slow, clunky, and almost impossible to develop a stable application with. Well, this has obviously all changed with 9.3 and I have blogged the praises of of it many times. In my eyes, 9.3 is truly a ground breaking product and allows for very rapid application application development with the Javascript or Flex (flash builder?) api. However, the most powerful aspect of ArcGIS Server hasn't been getting the attention it oh so deserves. Geoprocessing tasks. Most of the folks I talk to, arent even aware that you can use them. The ability to use Geoprocessing tasks were available at 9.2 by way of the ADF, but we realize not too many folks would touch them there, and rightfully so.

So, the conversation usually starts like this: "hey, you know all those cool models you have built to do all those complex tasks?" "Yes" "Well guess what...you can actually take those and publish them on the web" "No freakin way". Then I go on to explain about model builder and just how easy it is to publish them.

Take this example. You have a web application that needs to allow the user to easily find their parcel and then determine the exact types and percentages of the different soil types contained on that parcel. In ArcIMS, this was essentially undoable (ok, not without some heavy data preprocessing). Tasks such as these are trivial with ArcGIS Server. Open model builder, drag out a few tools (think clip and union) and some data sources, and viola, your are done. Publish that to ArcGIS Server, and there you have it. And that's a very straightforward example. Can you think of a tool that you may have created that could be pushed a to a wide, web based audience? I'll bet you can. Imagine the possibilities here. ESRI has done a fantastic job with this, yet I dont think that the general public (meaning Joe Shmoe ArcMap user) has any idea that ArcGIS Server can do this.

So, I put you on notice. Over the next few weeks, we are going to be blogging about setting up your own Geoprocessing Tasks, publishing them, and utilizing them in your applications (Flex and JS examples too).

We recently came across an issue with our queryTask and findTask functions when using apostrophes. If you submit a query or find task and the search text contains an apostrophe, you will get the following result:

Unable to perform query. Please check your parameters.

This is actually an issue with SQL Sever and not the Javascript API. To get SQL Sever to recognize the single apostrophe, you need to replace the single apostrophe with a pair of apostrophes.

To fix this issue, we used the following javascript code:

var county = dojo.byId('countySelect').value;

county = county.replace("'","''");

This should be done for any queryTask or findTask where the data being searched contains apostrophes.

I recently ran into some strange issues with FIND tasks in ArcGIS Server 9.3.1. This machine was a clean install and went from did 9.3 straight to 9.3.1. Everything looked great, (especially the speedy MSD files...another blog post on these coming soon) except our FIND tasks were broken. Nothing would return. No error, just an empty json response.

So, tried lots of different things...Different datasources (different flavors of SDE, file based GDB, etc) to no avail. Well this morning, I wiped the machine and did a reinstall and did 9.3 --> 9.3 sp1 --> 9.3.1 and the FIND task now works.

It could have been a pure coincidence, but to those that are upgrading or building a new box, it doesn't hurt to apply sp1 before moving to the 9.3.1 update. Usually, these things contain cumulative fixes, so I don't really have a good explanation. Special thanks to Ravi from the Redlands ArcGIS Server team for assisting me with this issue.

If you have ever worked with any ArcGIS product, you know all to well the concept of file locking. How many times have you tried to delete an old shapefile or person GDB, only to get the old 'File is being used by another process', yada yada. It always seems to happen to me when I am testing out a new Geoprocessing Tool or trying to delete an old mxd.

Well, Unlocker to the rescue. This handy little utility has saved me endless frustration when dealing with these file locking situations. Not only will it tell you what process is locking your file, it can 'unlock' that process as well so you can delete or rename that file. This gets installed with me wherever I go. Careful on the install though, by default it wants to install 'ebay shortcuts', which is a money maker for the author. I'd suggest a small donation instead.

The other tool that has come in handy in the past is one of the PSTOOLS...psexec. This little cmd line utility will allow you to launch a process as another user. Yes, you can also right click and choose 'run as' but this lets you create a batch file and run it over and over. Why would I want this? Well, I have had a few instances of having to connect to an ArcGIS server as another user to test permissions, etc, and this has been a handy tool to have.

Quick note for others that may run into this. Last week, we got the ArcGIS Server 9.3.1 update. I was excited to get this installed and tested after hearing about all of the speed improvements that have been promised. Anyway, long story short, got an error when trying to install.

"due to an error in software restriction policy processing..."

That was the gist of the error. Anyway, looking in event logs, this error appeared:

"The installation of F:\ArcGIS-Windows\ArcGISServerDotNet931.msp is not permitted due to an error in software restriction policy processing. The object cannot be trusted."

Looking that up, I was able to track down this patch from microsoft: http://www.microsoft.com/downloads/details.aspx?FamilyId=8EFFE1D9-7224-4586-BE2B-42C9AE5B9071&displaylang=en

That fixed the problem.

I had the strangest thing happen to me recently. I recently order and configured a new server for a client with SQL 2008. This was a beefy box. 8 gb of ram, dual quad core cpus, the works.

First impressions of SQL 2008 were good. Performance was good, and the SQL Server Studio had some nice usability enhancements that I appreciated. Anyway, this server supported 2 ArcSDE instances for a very high volume site. I set up SQL server to pretty much take as much memory as it wanted. I really wanted to cache as much as I could.

So, one day, I pop in and open up Process Explorer and see how we are looking. To my shocking disbelief, the server was reporting that it was almost maxed out on physical memory - right around 7.4 gbs. Wow I thought, thats a lot of RAM. Then I looked a bit closer, and the SQL Server process itself was only report 200 mb of RAM usage! Wha wha what? Where in the heck is all my memory going if SQL Server isnt using it? Again, using Process Explorer, I exported all my active processes and dumped into excel, where I added up all my process' RAM usage - and it only came to about 1.6 gbs. Where in the heck did remaining memory go?

Well, turns out, SQL Server 2008, unlike previous versions, does not report its total memory usage to task manager or to Process Explorer.

The amount of memory that will show up in Task Manager (or Process Explorer) for the SQL Server process is just what is used by the process itself. It does not represent the memory used by the SQL Server buffer pool. The SQL execution engine (SQLOS in SQL 2005+) manages its own threads and therefore takes care of its own memory allocations. The SQL command

DBCC MEMORYSTATUS

will provide a breakdown of the SQL memory clerks. In particular, take a look at the MEMORYCLERK_SQLBUFFERPOOL output, as this will tell you how much memory is being used by the buffer pool, which is likely where most of the system's memory is allocated.

For more information, see these links to the Microsoft Developer Network (MSDN) site about SQL Server Memory:

Memory Management Architecture - http://msdn.microsoft.com/en-us/library/cc280359.aspx Monitoring Memory Usage (This talks specifically about monitoring memory usage in SQL 2008. Provides some SQL-specific perfmon counters that show the actual memory usage) - http://msdn.microsoft.com/en-us/library/ms176018.aspx

In the end, I was able to use Perfmon to find all my missing memory. I used SQLServer:Memory Manager - Total Server Memory in PerfMon. Not sure I think that this is the smartest thing to do MS. I have a feeling this will freak out a good amount of folks...At least I know now though...Hope this helps someone and I can save them the 3 days of frantic Google Searches...

Using ESRI's Dynamically create layer list example, we created a simple, easy to use table of contents for the Javascript API. The toc allows users to zoom the extent of a service, expand or collapse layers, and of course turn layers on and off.

Check it out and view the source to see the code in action: ROK ArcGIS Server Table Of Contents Example

How it works: A function named "addToTOC" takes in two parameters. Layers is the actual layer being added to the map, and listLayers is a Boolean value that will either list the layers in the service or just at the layer at the root of the TOC.

The function checks to make sure the layer has been loaded to the map. If it has not been loaded, it adds an "onLoad" listener to the layer.

After confirming the layer has been added to the map, the layer is passed to one of two functions depending on the value of the listLayers parameter.

BuildLayerList:
Adds the layer to the root of the TOC and lists the sublayers in that service. These layers can then be turned on or off. The method is good for multilayer caches or dynamic services.

BuildLayerListRoot:
Only adds the layer to the root of the TOC. This method is good for layers hosted by ESRI and fused cache layers.

There are three functions that interact w/ the table of contents once it has been created:

ZoomToLayer:
Zooms to the full extent of the service
ToogleService:
Turns a service on or off
UpdateLayerVisibility:
Turns layer in a service on or off

Extra - Transparency Slider:

Finally, we have also added a transparency slider that works with the layers added to the map. As you switch layers from the pulldown list, the slider value is updated to the current opacity of the layer. This is pretty neat.
AddToTransparencyList:
Alphabetically adds the layer to the transparency pull down list
UpdateTransparencyLayer:
Updates the global variable that stores the transparency layer id and sets the slider to the opacity of the layer
ChangeTransparency:
Updates the opacity of the layer

We have found these functions to be quite useful, so we thought we would share these with the rest of the community. We hope you find them as useful as we do. If you have any trouble with them, post in the comments.Enjoy.

Hey all. Just a reminder, now the IE 8 is released, to check out your ArcGIS Server javascript api based sites and make sure they all work as you expect. I for one have had some issues with some graphic layers, so I have added a custom header to IIS instructing IE 8 to render pages in IE 7 compatibility mode.

Here is how to add the custom header to IIS (lifted from msdn):

To configure IIS 6 and earlier versions to include a custom HTTP response header, follow these steps:

1. Click Start, click Run, type inetmgr.exe, and then click OK.

2. Expand the server you want and then expand Web Sites.

3. Right-click the Web site you want and then click Properties.

4. Click the HTTP Headers tab.

5. Under Custom HTTP headers, click Add.

6. In the Custom header name box, type X-UA-Compatible.

7. In the Custom header value box, type IE=EmulateIE7.

8. Click OK two times.

Sure, I should have taken the time to test with the previous IE 8 release candidates, but you know the life of the developer is filled with other more important things, like keeping clients happy and getting my fix of Mt Dew for the day. So, I'll be using this method until I can properly figure out my graphics issues.

Also, another tool that I have been meaning to blog about is IEtester. It comes with 'My DebugBar' which is the IE equivalent of Firebug. Well, I shouldn't even say that, its not anywhere near as functional as Firebug. Anyway, IEtester lets you view your sites in different IE rendering engines. From 5.5 all the way to 8. Its been a pretty handy tool during testing. Enjoy.

PS: Anyone have an extra Asheville Phish ticket?

We have a high visibility site getting ready to launch - and we have been wondering about an issue for a while now. Up until now, it took a backseat to other issues while in development. But now that its officially launch time, we decided to dig a bit deeper and figure out how to get every last bit of performance out of the app.

The site is using a multi layer cache. This site uses lots and lots of different layers. They are also nationwide - and then all the way down to zipcode level. The application needs to be able to turn on and off lots of different layers and switch between many map services. For this reason, the best option for us was to use the Multi Layer Cache. The data itself is pretty static, so caching was the way to go here. If we were to try and draw this dynamically, it would quickly overwhelm our servers as this is a very high visibility site and will be taking lots of hits.

So, when we got to actually writing the application, we used (surprise surprise) the ArcGIS Server JavaScript API. Using the oh-so-cool REST services explorer (wouldn't this have been nice with ArcIMS?), the map service shows up as Single Fused Map Cache: false. No mention of a Multi Layer cache. So, the only way to utilize these multilayer cache services in our application is via ArcGISDynamicMapServiceLayer.

So, what does that mean? Is this just a bug? Is there a reason to cache these at all? Well, the answer is yes. While the JS API still consumes these services as a ArcGISDynamicMapServiceLayer, the server component, when assembling the map, does not have to re-render the map each time. It simply assembles the tiles on the server and then sends back the completed image. While this is not exactly ideal, or what I would have expected, its much better than the server having to render the map on each request and then export/send it to the client.

We did some informal testing and using the multilayer cache dynamic request vs pure dynamic request and the speed gains were quite significant.

Moral of the story? If you have a need for a multilayer cache, use it. There are huge speed advantages to using it over a pure dynamic layer. If you can get away with a fused cache, that is obviously the better choice, because it will stream the pre rendered tiles instead of a map image.

I have heard from some folks at ESRI that the upcoming js api v1.3, we may have some other ways of doing things and taking better advantage of these types of caches. We'll be sure to blog about this further when its out.

Well, after much debate, I decided to go with the Dell monster. Each time I spec out a new sever, I am amazed at how much horsepower you can get for your money these days. Hardware prices, most notably RAM and disk, have absolutely plummeted.

I picked up a PowerEdge 2900 with dual quad core cpus. 8 gb of ram and a small array for the operating system. I chose to add a second array to store the cache directly on the machine. Again, hard drive space is so cheap these days...

So, long story short, this thing absolutely crushes map cache. I can throw 8 instances at it and it will easily create the cache 10 times faster than I am used to. We have several nationwide level caches that get updated regularly, so this new addition has been very welcomed...

Now that the Arcgis Server javascript and flex apis have really taken off, the demand for map caches is growing. We have a couple of machines that we use for developing Arcgis server based applications, and one big, dedicated production box.

So, when it comes time to build map cache, the question always arises...Where should we build it? On the production box? Should we throttle the processes? On the development boxes? Our development boxes are usually the older, hand me downs (from production) and are usually slower by today's standards.

So, we are always stuck. If we build cache on the prod box, we run the risk of slowing down production applications. If we build in our development boxes, it will take forever. We have gotten to the point where we are creating new caches 24 hours a day, 7 days a week.

So, I sit here, wondering how to make the best of my hardware budget. I could go to dell, and drop 5k on a monster with 16gb of Ram and dual 3 ghtz quad core cpus. But do I need a machine like that? Those boxes are huge with all sorts of space for disk arrays, etc - and the price tag reflects that. This has me really thinking like the cloud computing infrastructure is fleshing out. Like how google specs out their servers for pure high density computing. All I need is pure CPU madness to crank out cache as fast as possible. Well, that and some Ram, which is so cheap these days. I have a nice SAN to put all the cache, so I should be set.

What would you do? Build your own cache server with the latest and greatest cpu? Invest in a blade system? Put it all in the Cloud?