Category Archives: data

New Additions to the Collection, May 22, 2017

The following titles were recently added or updated.

CU Numeric Data Catalog Holdings

New

CU Spatial Data Catalog Holdings

New:

New Additions to the Collection, May 15, 2017

The following titles were recently added or updated.

CU Numeric Data Catalog Holdings

New

Recently updated

CU Spatial Data Catalog Holdings

New:

Survey Documentation and Analysis (SDA)

Survey Documentation and Analysis (SDA) is a web based interface that allows access and analysis of data. The data can be accessed from IPUMS or from  the Inter-university Consortium for Political and Social Research (ICPSR).

SDA allows you to:

  • Browse the codebook describing a datasetsda
  • Calculate frequencies or crosstabulation (with charts)
  • Do comparison of means
  • Calculate a correlation matrix
  • Compare correlations
  • Perform multiple regression
  • Perform logit/probit regression
  • List values of individual cases
  • Recode variables (into public work area)
  • Compute a new variable
  • List/delete derived variables
  • Download a customized subset

SDA allows you to analyze data at a level appropriate to your level of experience. SDA can only analyze datasets that reside on an SDA server. If you would like to test drive SDA, or to see if SDA is useful for your research check out their General Information page.

NHGIS Releases Standardized Tables for 2000 & 2010 Census

Comparing Census data from one decade the next can be difficult since boundaries  and variables can change. Trying to figure out those differences can be time consuming and sometimes frustrating. To help make this easier, NHGIS recently released 65 time series tables standardizing 2000 & 2010 100% count variables to the 2010 Census boundaries for ten geographic levels including tracts, block groups and 5 digit zip code tabulation areas (ZCTAs).

The tables are marked with a TS from within the Topic filter

NHGIS topic filter

Available standardized tables are listed in the Time Series Tables tab, and labeled Standardized to 2010 under the Geographic Integration column.

nhgis_tables

More information on time tables produced by the NHGIS can be found in the Data Documentation section, including both the geographically standardized tables as well as nominally integrated tables. The page includes methodology, and a PDF containing the complete list of variables within the time series datasets.

NYC Geocoding

NYC Dept of City Planning offers a few options for geocoding including a stand alone application called Geosupport, which personally, I don’t find the easiest to use, and a Geoclient API. But for geocoding in ArcGIS quickly, or if you have hundreds of thousands, or even millions of addresses I find LION to be the best option. The only problem, as of release 12C, the geodatabase no longer includes the locators for borough & zip. The current version is 15A,

We’ve taken the old locators from 12C and imported them into the latest version which seems to work pretty well. This is available on the spatial data on the internet page, and we generally update this with each new release.

Mapping Voting Rights Act Section Five Locations

This summer, a couple of us in the DSSC have decided to sit down and learn Leaflet & D3. We'll post some of the maps we make in the blog throughout the summer.

I'm grateful to Peter Leonard, the Librarian for Digital Humanities Research at Yale for getting me started on Leaflet at the NYPL Maps Hack 2013 organized by NYPL Labs.

As an exercise, I decided to map out the places no longer covered by the recent Supreme Court decision (PDF) on section four of the Voting Rights Act.

I took the tables on the Dept of Justice section 5 covered jurisdictions page, joined them together with boundaries from the US Census Bureau's TIGER/Line, simplified the shapefiles in QGIS (removed several fields too), exported to GeoJSON and brought the layers into Leaflet.

The map is sitting at a temp location until the bugs get worked out and we can include the maps in CU Spatial blog posts.

As you can see, it's fairly straight forward and includes some modified (simplified?) code from tutorials on the Leaflet site. The color choices came from ColorBrewer.

One thing I think would improve the map a bit would be to be able to click on the word 'townships' in the legend and have it automatically zoom to Michigan since at the starting scale, both townships are difficult to see (same with the three NYC counties).

Eventually, I'll redo this map in D3 because it'll allow me to use a projection better suited for North America.

Eventually we'll put the data into the Spatial Data Catalog, but for now I'm including a link to zip file containing both the shapefiles & geojson layers without proper metadata and a "use at your own risk" disclaimer. If you use it and find any mistakes, please let us know!

Using OpenStreetMap XML data

osm_xml2

OpenStreetMap is a great resource mostly user created and maintained. The data is free of charge and there are several ways of accessing the data for use in a GIS project.

One of the easiest and quickest ways is to download data from CloudMade in shapefile format, but occasionally not all of the features shown in OpenStreetMap are available in the shapefile download.

An alternative is to download the OSM XML data, open it in QGIS, and if needed, export to shapefile.

To do this, open QGIS and under the Plugins menu, select Manage Plugins and turn on the OpenStreetMap plugin. If it’s not there then you will have to add it from Fetch Python Plugins.

The plugin allows for viewing downloaded OSM XML data, downloading large scale areas directly, and uploading edits you’ve made (account required).

Click on the Load OSM from file icon and navigate to the downloaded OSM XML layer.

Put a check mark next to the fields you want to create (these will only populate if the information is encoded). Keep a check mark next to Use custom renderer if you want to symbolize your data similar to the OSM scheme.

If you need to work with any of the features in shapefile format, right click on a layer and select Save vector layer as, and choose ESRI shapefile from the Format pull down menu if not already selected.

And that’s it!

NYPL Map Rectifier

nyplmap

 NYPL has a new toy tool for rectifying digitized historic maps from their collection called the NYPL Map Rectifier.

After creating an account, browse through the various maps and collections, choose a map to rectify, clip or mask the image, and finally export out to either a KML (for Google Earth) or a WMS base URL for use in a GIS software. Some maps have the option to export into either PNG or TIF formats as well.

There is also a vectorize option for a few of the rectified maps (I didn’t see this option on most maps, I’m guessing because it’s still in beta). For these, the vector layers can be downloaded in several formats including shapefile.

NYPL has also created a site called Relief Map Warper and has uploaded maps for Haiti which after georeferencing, the maps will become available in Open Street Map for tracing, and also made available for use on the ground by relief workers. I highly recommend joining and helping out if you can.

 

 

New Spatial Data

Thought we would mention a few new spatial datasets in our collection. We now have updated Transportation layers for New York city including routes and lines for subways, buses, freight and commuter rail as well as updated truck routes and zones.

 

Also, we have added New York city police precincts and crime data for 2001, 2005 and 2006.

 

 

From ESRI we now have new raster datasets in worldwide coverage representing elevation, shaded relief and Landsat imagery.  These datasets can be accessed in the DSSC facilities via the ArcSDE server.